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EXECUTIVE  SUMMARY 


The  first  United  States  Joint  Forces  Command  (USJFCOM)  Multinational 
Limited  Objective  Experiment  (MN  LOE)  I  was  conducted  31  October  -  16  November  2001. 

MN  LOE  I  was  the  first  of  a  series  of  Joint  Futures  Lab/J9  LOEs  to  be  conducted  with 
multinational  partners  to  define  coalition  participation  in  Olympic  Challenge  2004.  The 
participants  in  MN  LOE  I  -  Australia  (AUS),  Canada  (CA)  (as  an  observer),  Germany  (GE),  the 
United  Kingdom  (UK),  and  the  United  States  (US)  -  are  all  members  of  the  Multinational 
Interoperability  Council  (MIC).  The  experiment  was  conducted  in  a  distributed  environment;  the 
US  planning  cells  were  located  at  the  Joint  Battle  Center  in  Suffolk,  Virginia  and  the  Australian, 
UK,  German,  and  Canadian  cells  were  located  in  Fem  Hill  Park,  Portsdown  West,  Potsdam,  and 
Ottawa,  respectively. 

MN  LOE  I  was  comprised  of  seven  independent  planning  sessions.  Each  session 
addressed  a  situation  or  vignette  for  which  the  multinational  coalition  conducted  Mission 
Analysis  and  developed  a  Course  of  Action.  Given  that  MN  LOE  I  was  an  unclassified 
experiment  the  vignettes  developed  for  the  experiment  were  restricted  to  Military  Operations 
Other  Than  War  (MOOTW),  in  particular  humanitarian  operations. 

The  strategy  adopted  for  examining  collaborative  planning  in  MN  LOE  I  was  to 
have  two  planning  teams,  Traditional  and  Integrated,  conduct  planning  concurrently.  The 
Traditional  planning  team  used  a  process  based  on  the  Lead  Nation  concept  described  in  the 
Multinational  Interoperability  Working  Group  (MIWG)  white  paper  whereas  the  Integrated 
Planning  Team  used  an  experimental  process  based  on  Rapid  Decisive  Operations  (RDO) 
Standing  Joint  Command  and  Control  Element  (SJC2E)  concepts.  Both  teams  focused  on 
Mission  Analysis  (MA)  and  Course  of  Action  (COA)  development. 

Three  Critical  Operational  Issues  (COIs)  were  examined  in  MN  LOE  I: 

COI  #1:  With  the  introduction  of  collaborative  tools,  did  the  Traditional  and  Integrated 
planning  processes  evolve  into  similar  processes  [from  the  perspective  of  multinational 
collaboration]  or  were  they  different? 

COI  #2:  Assuming  both  planning  processes  are  equipped  with  the  same  collaboration 
capability,  does  the  quality  of  the  CO  As  produced  by  the  Traditional  and  Integrated 
planning  processes  differ  as  a  function  of  length  of  planning  window  and  type  of 
operation? 

COI  #3:  Given  its  distributed  structure,  did  MN  LOE  I  demonstrate  that  collaborative 
planning  can  be  conducted  successfully  in  a  distributed  environment? 
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Significant  findings  for  the  three  COIs  were: 


COI  #1: 

•  The  Traditional  and  Integrated  processes  followed  in  MN  LOE  I  demonstrated  significant 
similarities  as  well  as  differences,  and  consequently  a  definitive  comparison  of  the  two 
processes  proved  to  be  challenging.  The  processes  did  exhibit  noteworthy  differences  in 
the  areas  of  leadership  and  situational  awareness.  Issues  related  to  leadership  have 
potential  implications  for  doctrine,  procedures  and  training. 

•  Largely  due  to  the  collaboration  tools,  the  Traditional  process  followed  in  MN  LOE  I  was 
more  integrated  in  nature  than  the  conventional  ‘stove-piped’  approach  typically 
associated  with  the  Lead  Nation  concept.  Further  investigation  is  required  to  assess 
whether  given  the  availability  of  a  robust  collaboration  environment  and  a  commonly 
defined  product,  the  Traditional  process  might  naturally  evolve  into  something  similar  to 
the  Integrated  process. 

•  The  infrastructure  implications  related  to  maintaining  continuous  situational  awareness  of 
team  activity  merits  further  study.  System  and  network  performance  data  collected 
during  MN  LOE  I  should  prove  very  useful  in  this  endeavor. 

•  By  design,  planning  during  MN  LOE  I  was  primarily  conducted  at  the  operational  level. 

It  would  be  of  value  to  examine  the  adaptation  of  the  Integrated  process  to  the  tactical 
level  (i.e.,  more  detailed,  real-time  planning). 

COI  #2: 

•  A  panel  of  senior  subject  matter  experts  (the  “Graybeards”)  who  evaluated  the  COAs 
generated  during  MN  LOE  I  marginally  preferred  the  COAs  produced  by  the  Integrated 
team,  citing  completeness  as  a  critical  factor.  Given  the  small  size  of  the  graybeard  panel 
(five),  statistical  significance  cannot  be  assigned  to  most  of  the  individual  vignette  scores. 
Two  exceptions  are  Vignettes  E  and  H,  where  a  significant  majority  of  the  experts 
selected  the  COAs  produced  by  the  Integrated  team. 

•  Detailed  analysis  of  the  panel’s  evaluations  did  not  provide  a  definitive  explanation  of  the 
facto rs(s)  which  caused  the  graybeards  to  prefer  the  Integrated  COAs.  The  examination 
of  the  types  of  operations  and  lengths  of  planning  window  represented  in  the  seven 
vignettes  did  not  suggest  an  overall  trend  that  would  readily  explain  the  graybeard 
selections. 

•  Potential  explanations  of  the  panel’s  evaluations  that  merit  further  investigation  are: 

■  The  Integrated  process  may  be  better  suited  to  low  intensity  operations  (i.e., 
humanitarian)  such  as  those  examined  in  the  LOE. 

■  The  Integrated  team  had  the  advantage  of  pre-LOE  training  in  a  process  that  remained 
relatively  stable  throughout  the  LOE. 

■  The  Integrated  team  was  able  to  better  retain  corporate  knowledge  and  leverage  off 
previous  planning  efforts. 

COI  #3: 

•  MN  LOE  I  demonstrated  that  collaborative  planning  can  be  conducted  in  a  distributed 
environment. 
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•  Even  with  this  positive  result,  the  potential  adverse  impact  of  computer  system  problems 
and  network  congestion  on  the  planning  processes  should  be  neither  ignored  nor 
minimized.  Additional  study  that  leverages  off  insights  gathered  during  MN  LOE  I  is 
required  to  develop  a  better  understanding  of  the  system  requirements  associated  with 
supporting  distributed  planning,  especially  at  the  level  represented  in  the  Integrated 
process.  Likewise,  additional  study  is  required  to  address  the  unique  management  issues 
associated  with  conducting  a  LOE  in  a  distributed  environment. 

•  By  design  the  exchange  of  classified  data  was  not  addressed  in  MN  LOE  I,  but  will  need 
to  be  explored  in  future  LOEs. 

MN  LOE  I  was  unique  since  it  was  a  focused  discovery  experiment  structured  to 
examine  a  specific  aspect  of  the  RDO  concept,  distributed  planning  with  multinational  partners. 
MN  LOE  I  was  also  a  pilot  event  in  that  it  attempted  a  global  distributed  experiment  linking  sites 
that  had  never  been  linked  using  a  relatively  new  collaboration  tool  and  approach.  The  MN  LOE 
took  an  innovative  approach  for  controlling  the  experiment  by  relying  on  a  distributed  virtual 
control  cell  in  lieu  of  the  more  traditional  centralized  approach.  Finally,  the  MN  LOE  served  as 
a  venue  for  focused  research  topics  of  national  interest.  The  feedback  collected  from  participants 
and  observers  of  MN  LOE  I  and  the  attendees  of  the  Analysis  Workshop  provided  The  Johns 
Hopkins  University  Applied  Physics  Laboratory  (JHU/APL)  analysis  team  with  many  valuable 
recommendations  that  will  ultimately  benefit  future  LOEs.  The  detailed  discussion  of 
recommendations  provided  in  Section  4  includes  the  following  key  recommendations: 

•  Having  a  point  of  participation  in  each  nation  facilitated  the  availability  of  national  staff 
to  participate  in  the  MN  LOE.  This  coupled  with  the  insight  gained  about  multinational 
planning  in  a  distributed  environment  make  a  compelling  argument  for  designing  future 
MN  LOEs  with  a  similar  distributed  structure. 

•  Several  topics  worthy  of  additional  investigation  were  identified  during  the  analysis  of 
MN  LOE  I.  It  is  recommended  that  J9  consider  the  following: 

■  Given  the  availability  of  a  robust  collaboration  environment  and  a  commonly  defined 
process,  might  the  Traditional  process  naturally  evolve  into  something  similar  to  the 
Integrated  process  and  if  not,  what  are  the  major  obstacles?  In  particular,  the  role  of 
the  lead  nation  as  a  leader  needs  to  be  explored  further  in  the  Integrated  process. 

■  What  are  the  infrastructure  implications  associated  with  maintaining  continuous 
situational  awareness  of  team  activity  as  demonstrated  in  the  Integrated  process? 

•  The  MN  LOEs  are  critical  to  examining  and  demonstrating  the  value  and  viability  of 
proposed  RDO  concepts.  If  a  comparison  of  new  concepts  against  more  ‘traditional’ 
concepts  is  not  part  of  the  LOE  structure,  then  a  comparative  assessment  will  need  to  be 
obtained  through  some  other  means. 

•  A  planning  roadmap  for  Olympic  Challenge’04  that  includes  the  MN  LOEs  is  critical  to 
ensuring  that  the  objectives  of  the  individual  MN  LOEs  support  the  larger  objectives 
defined  for  Olympic  Challenge  ’04.  Also,  it  is  of  paramount  importance  to  start  planning 
for  the  MN  LOEs  as  soon  as  possible.  An  early  start  is  necessary  to  ensure  that  critical 
issues  (e.g.,  migration  to  a  classified  environment,  staffing  requirements,  1-  vs.  2-process 
structure)  are  addressed  early  in  the  process,  thus  increasing  the  potential  for  successful 
resolution. 
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Section  1 

INTRODUCTION 


1.1  INTRODUCTION 


The  first  United  States  Joint  Forces  Command  (USJFCOM)  Multinational 
Limited  Objective  Experiment  (MN  LOE)  I  was  conducted  31  October  -  16  November  2001. 

MN  LOE  I  is  the  first  of  a  series  of  Joint  Futures  Lab  (J9)  LOEs  conducted  with  multinational 
partners  to  define  coalition  participation  in  Olympic  Challenge  2004.  The  participants  of  MN 
LOE  I,  Australia  (AUS),  Canada  (CA),  Gennany  (GE),  the  United  Kingdom  (UK),  and  the 
United  States  (US),  are  all  members  of  the  Multinational  Interoperability  Council  (MIC).1 

Section  1  of  this  report  describes  the  objectives,  experiment  design,  evaluation 
strategy,  and  data  collection  strategy  defined  for  MN  LOE  I.  Section  2  provides  an  overview  of 
the  events  that  occurred  during  the  initial  training  session  and  the  two  weeks  of  planning  sessions 
comprising  the  LOE.  MN  LOE  I  analysis  strategy  and  findings  are  detailed  in  Section  3,  and 
Section  4  provides  a  summary  of  recommendations  compiled  during  the  experiment  and  the 
subsequent  analysis  phase  that  are  relevant  to  the  planning  of  future  LOEs. 


1.2  MN  LOE  I  OBJECTIVES 


One  of  the  key  objectives  of  MN  LOE  I  was  for  the  multinational  participants  and 
J9  to  successfully  design  and  conduct  a  multinational  experiment.  An  equally  critical  objective 
of  MN  LOE  I  was  to  explore  collaboration  with  coalition  partners  during  Rapid  Decisive 
Operations  (RDO)2  planning.  The  strategy  adopted  for  examining  collaborative  planning  in  MN 
LOE  I  was  to  have  two  planning  teams,  Traditional  and  Integrated,  conduct  planning 
concurrently.  The  Traditional  planning  team  would  use  a  process  based  on  the  Lead  Nation 
concept  described  in  the  Multinational  Interoperability  Working  Group  (MIWG)  white  paper3 
whereas  the  Integrated  Planning  Team  would  use  an  experimental  process  based  on  RDO 
Standing  Joint  Command  and  Control  Element  (SJC2E)  concepts.4  Both  teams  would  focus  on 
Mission  Analysis  (MA)  and  Course  of  Action  (COA)  development. 


1  Canada  sent  observers  to  MN  LOE  I.  France,  also  a  member  of  the  MIC,  did  not  participate  in  MN  LOE  I. 

2  “Rapid  Decisive  Operations  is  a  joint  operational  concept  for  future  operations.  A  rapid  decisive  operation  will 
integrate  knowledge,  command  and  control,  and  effects-based  operations  to  achieve  the  desired  political/military 
effect.  In  preparing  for  and  conducting  a  rapid  decisive  operation,  the  military  acts  in  concert  with  and  leverages 
the  other  instruments  of  national  power  to  understand  and  reduce  the  adversary’s  critical  capabilities  and 
coherence.  The  United  States  and  its  allies  asymmetrically  assault  the  adversary  from  directions  and  in 
dimensions  against  which  he  has  no  counter,  dictating  the  terms  and  tempo  of  the  operation.  The  adversary, 
suffering  from  the  loss  of  coherence  and  unable  to  achieve  his  objectives,  chooses  to  cease  actions  that  are  against 
US  interests  or  has  his  capabilities  defeated.”  U.  S.  JFCOM  J-9  Joint  Futures  Laboratory,  A  Concept  for  Rapid 
Decisive  Operations  (Draft),  RDO  Whitepaper  Version  2.0,  pp.  ii 

3  “MIWG  Report  to  the  Multinational  Interoperability  Council  Topic:  The  Lead  Nation  Concept  in  Coalition 
Operations,”  20  December  2000. 

4  J9  training  materials  for  the  RDO  SJC2E  concepts  are  provided  in  Appendix  A. 
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1.2.1  MN  LOE  I  CRITICAL  OPERATIONAL  ISSUES 

During  final  planning  of  MN  LOE  I  it  was  decided  to  allow  the  Traditional 
process  to  adapt  freely  to  the  collaborative  environment  and  not  constrain  the  process  by  static, 
common  operating  procedures.  Consequently,  COI  #1  was  factored  into  the  data  collection  and 
analysis  effort  to  track  the  evolution  of  the  processes  over  the  course  of  the  experiment: 

COI  #1:  With  the  introduction  of  the  collaborative  tool,  did  the  Traditional  and  Integrated 
planning  processes  evolve  into  similar5  processes  [from  the  perspective  of  multinational 
collaboration]  or  were  they  different? 

The  second  COI  formulates  the  comparison  of  the  two  planning  processes 
examined  in  the  LOE. 

COI  #2:  Assuming  both  processes  are  equipped  with  the  same  collaboration  capability,  does 
the  quality  of  the  COAs  (completeness,  accuracy  and  suitability)  produced  by  the 
Traditional  and  Integrated  planning  processes  differ  as  a  function  of  length  of planning 
window  and  type  of  operation?  Specifically,  does  one  process  produce  a  better  product 
(i.e.,  Course  of  Action)  when  planning  time  is  short? 

During  MN  LOE  I  the  national  planning  cells  operated  from  sites  in  their  own 
countries:  Joint  Battle  Center  (JBC)  (US),  Portsdown  West  (UK),  Potsdam  (GE)  and  Fern  Hill 
Park  (AUS).  Given  its  distributed  structure,  MN  LOE  I  provided  an  excellent  opportunity  to 
explore  the  following  Critical  Operational  Issue: 

COI  #3:  Can  collaborative  planning  be  conducted  successfully  in  a  distributed 
environment? 


1.3  EXPERIMENT  DESIGN 

1.3.1  SCENARIO  AND  VIGNETTES 

MN  LOE  I  was  conducted  as  a  series  of  independent  planning  sessions.  Each 
session  addressed  a  situation  or  vignette  for  which  the  multinational  coalition  conducted  Mission 
Analysis  and  developed  a  COA.  The  initial  step  in  developing  the  vignettes  for  MN  LOE  I  was 
to  identify  potential  operations  and  geographic  areas  to  be  used  as  the  framework  for  each 
vignette.  All  vignettes  developed  for  MN  LOE  I  assumed  the  operational  environment  defined 
in  the  Pacifica  scenario6.  Given  that  MN  LOE  I  was  an  unclassified  experiment,  the  vignettes 


5  The  planning  processes  are  complex  and  the  assessment  of  ‘similar’  versus  ‘different’  is  based  on  a  set  of 
characteristics  common  to  both  processes:  Organizational  Structure,  Use  of  Tool,  Schedule  of  Events,  Role  of 
Lead  Nation,  Impact  of  Lead  Nation  on  Process,  Effectiveness  of  Inter-Country  Interactions,  Effectiveness  of 
Intra-Country  Interactions,  and  Balance  in  Skills/Experience/Background  of  Team  Members.  A  detailed 
comparison  of  the  two  processes  demonstrated  during  MN  LOE  I  is  provided  in  Section  3 :  Analysis  Results. 

6  Summary  of  Pacifica  scenario:  (South  Pacific)  The  country  of  Pacifica  is  depicted  as  a  vital  and  important  trading 
and  security  partner  of  the  United  States.  Pacifica  is  presently  experiencing  a  civil  war  that  is  concentrated  on  the 


1-2 


JWR-02-007 


MNLOEI 


developed  for  the  experiment  were  restricted  to  Military  Operations  Other  Than  War 
(MOOTW)* * * * * 7,  in  particular  humanitarian  operations.  The  military  operations  defined  for  Pacifica 
served  as  a  background,  ongoing  Joint  Task  Force  effort,  while  the  contingencies  defined  for  the 
vignettes  spanned  the  spectrum  of  MOOTW.  Countries  within  1200  nm  of  the  Philippines 
(fictitious  Pacifica)  were  identified  as  candidate  locations.  An  analysis  of  the  situations  allowed 
under  MOOTW  and  political  sensitivities  associated  with  the  South  Pacific  region  (based  on 
feedback  from  LOE  participants)  motivated  the  design  team  to  exclude  certain  operation/country 
combinations  from  consideration. 

Additional  guidelines  regarding  vignette  development  specified  that  all  vignettes 
should  be  of  comparable  complexity  and  represent  unique  operational  situations  that  do  not 
leverage  off  previous  vignettes.  These  requirements  were  intended  to  ensure  that  evaluation 
results  from  all  planning  sessions  could  be  combined  on  an  equal  basis  to  develop  an  overall 
comparison  of  the  two  planning  processes.  If  vignettes  represented  comparable  planning 
challenges,  the  different  length  planning  windows  ranging  from  6  to  10  hours  would  provide 
valuable  insight  into  the  quality  of  COA  product  as  a  function  of  length  of  planning  window. 

1.3.2  MN  LOE  I  ARCHITECTURE 

The  network  architecture  developed  for  MN  LOE  I  is  depicted  in  Figure  1-1. 
Collaborative  planning  was  conducted  over  the  classified  Combined  Federated  Battle  Lab 
Network  (CFBLNet)8,  denoted  in  the  diagram  as  a  ‘cloud’  linking  the  five  LOE  sites.  The  US 
planning  cells  and  the  LOE  white  cell9  were  located  at  the  Joint  Battle  Center  in  Suffolk, 
Virginia.  The  Australian,  UK,  German,  and  Canadian  cells  were  located  in  Fern  Hill  Park, 
Portsdown  West,  Potsdam,  and  Otttawa,  respectively. 


island  of  Luzon,  and  an  insurgency  that  began  on  the  island  of  Mindanao  and  over  time  has  spread  throughout  the 

country.  The  current  scenario  includes  the  events  leading  to  the  deployment  of  US  Special  Operations  Force 

(SOF)  training  teams  to  assist  the  Republic  of  Pacifica  (ROP)  in  its  counterinsurgency  effort.  It  also  provides 

background  information  leading  to  the  eventual  employment  of  a  US-led  multinational  task  force  committed  to 

defeat  an  armed  invasion  of  Pacifica  by  neighboring  Surran. 

7  Operations  specified  in  Joint  Pub  3-07,  “Joint  Doctrine  for  Military  Operations  Other  Than  War.” 

s  The  CFBL  Initiative  is  a  consortium  developed  between  the  US,  NATO  and  CCEB  Nations.  “The  CFBLNet  is  a 
longer-term  combined  RDT&E  network.  It  is  not  intended  to  be  a  combatant  network  or  the  architecture  for  a 
future  network.  This  network  is  projected  to  remain  in  operation  to  conduct  coalition  C4I  experiments  and 
provide  possible  parallel  use  in  C1NC  coalition  exercises.  Creation  of  the  CFBLNet  leverages  Joint  Warrior 
Interoperability  Demonstration  (JWID)  resources,  existing  US  Federated  Battle  Laboratories  assets  and  coalition 
battle  laboratories/test  beds.  As  such  it  will  not  be  a  solely  US  owned/operated  network,  but  a  combined  network 
with  the  members  having  equal  say  in  its  utilization  and  management.  NATO  nations  participating  in  bilateral  or 
multilateral  CFBLNet  project  arrangements  are  responsible  for  Rinding  their  portion  of  the  effort  and  there  is  no 
requirement  for  monies  to  be  exchanged  among  participants.  The  NATO  point  of  entry  for  CFBLNet  is  NC3A 
The  Hague.”  [Joint  Warrior  Interoperability  Demonstration  2002  website, 
http://  www .  i  wid .  i  s .  mi  1/html/c  fb  lnet .  html .  1 

CFBLNet  was  chosen  for  MN  LOE  I  because  it  is  the  only  established  and  accredited  wide  area  network  available 
for  experimentation  with  multinational  partners. 

Q  The  white  cell  was  comprised  of  a  team  operating  as  the  staff  of  the  Supreme  Allied  Commander  Asia  Pacifica 
(SACAPAC),  the  analysis  team  led  by  JHU/APL  and  technical  personnel  monitoring  network  and  system 
performance. 
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Figure  1-1:  MN  LOE  I  Network  Architecture 


Both  planning  teams  at  each  site  were  equipped  with  Groove,  a  commercial 
collaboration  product  that  supports  peer-to-peer  communications  and  provides  centralized 
system  management.10  Pre-LOE  expectations  regarding  the  use  of  Groove  spaces  are  depicted  in 
Figure  1-2.  Each  participant  had  access  to  Groove  from  his  or  her  workstation.  Each  national 
planning  cell  was  assigned  its  own  planning  environment  (i.e.,  shared  space)  while  common 
shared  spaces  were  established  for  the  Integrated  team.  Prior  to  the  LOE  it  was  envisioned  that 
the  Traditional  national  cells  (US,  UK,  AUS,  and  GE)  would  operate  primarily  in  their  national 
spaces,  with  the  lead  country  selectively  inviting  other  countries  into  its  space.  The  Integrated 
team  would  operate  in  a  central  shared  space  as  well  as  shared  spaces  dedicated  to  planning, 
intelligence,  and  logistics  activities.  How  the  teams  actually  used  their  spaces  during  the  LOE  is 
addressed  in  Sections  2  and  3. 


10  Groove  product  information  available  at  http://www.groove.net. 
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(US  assumed  Lead  Nation) 


ft 


A  country  can  elect  to  invite/uninvite  other  player(s)  into  their  Groove  space. 


ft 


Process  business  rule  :  All  countries  are  invited  into  the  Integrated  Planning  spaces. 


Figure  1-2:  Anticipated  Uses  of  Groove  Spaces 


The  structures  of  the  shared  spaces  were  established  prior  to  the  LOE  and  were 
not  modified  by  the  participants.  All  shared  paces  provided  the  following  tools:  Notepad, 
Threaded  Discussion,  Agenda/Outliner,  Whiteboard,  Text  Chat,  Messaging  and  Audio.  In 
addition  to  collaborative  planning  among  team  members,  the  shared  spaces  also  supported 
information  exchanges  between  the  teams  and  the  Supreme  Allied  Commander  Asia  Pacific 
(SACAPAC). 


Each  national  cell  also  had  unclassified  access  to  the  World  Wide  Web  (www)  to 
conduct  open  source  research.  Each  cell  had  an  unclassified  workstation  equipped  with 
BrainEKP  (formerly  called  Team  Brain),  Autonomy  and  other  www  browsers.  In  lieu  of  a  true 
Operational  Net  Assessment  for  the  geographic  areas  specified  in  the  vignettes,  a  relational 
database  was  constructed  inside  BrainEKP1 1  that  provided  background  data  for  these  areas.  A 
manual  data  mining  effort  on  the  unclassified,  open  source  www  resulted  in  approximately  500 
URLs  for  ten  South  Pacific  countries.  These  URLS  were  catalogued  by  the  five  PMSEI 
(Political,  Military,  Social,  Economic,  and  Infrastructure)  categories.  Autonomy  Portal-in-a- 


1 1  “BrainEKP  is  composed  of  four  key  components: 

•  Universal  Data  Access  -  enables  the  integration  of  information  from  many  sources 

•  Integrated  Collaboration  -  enables  communication  in  the  same  place  where  information  is  created,  stored,  and 
accessed 

•  Knowledge  Model  -  enables  a  shared  understanding  of  how  everything  is  connected,  accessed,  processed,  and 
used 

•  Visual  User  Interface  -  enables  people  to  easily  use  and  navigate  the  system,  and  see  how  everything  fits 
together  in  context  “('http://www.thebrain.com/products/brainEKPl. 
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Box  '  provided  an  additional  means  of  conducting  focused  open  source  research  for  the  countries 
and  situations  considered  in  the  vignettes. 

1.3.3  VIGNETTE  SCHEDULE 

The  LOE  schedule,  detailed  in  Figure  2-1  (Section  2),  included  eight  vignettes 
(Vignettes  C  through  J)  that  ranged  from  6  to  10  hours  .  Two  additional  vignettes  were 
included  in  the  training  session  conducted  the  week  prior  to  the  LOE.  At  the  beginning  of  each 
planning  session  SACAPAC  issued  a  warning  order  that  summarized  the  operation  and  the 
objectives  to  be  addressed  in  the  COA.  The  warning  order  also  specified  times  when  the 
Mission  Analysis  products  (primarily  a  revised  Mission  Statement)  and  the  Course  of  Action 
were  to  be  delivered  to  SACAPAC.  The  teams  could  initiate  contact  with  SACAPAC  whenever 
they  chose  and  during  the  second  week  SACAPAC  introduced  periodic  updates  (“injects”)  to  the 
operational  situation.  At  the  end  of  each  vignette  the  participants  completed  evaluation 
questionnaires.  These  questionnaires  solicited  feedback  about  the  COAs  produced  during  the 
vignette  and  the  planning  process  that  was  used,  as  well  comments  pertaining  to  LOE  experiment 
design  and  structure.  Participants  had  an  opportunity  to  update  their  responses  the  following 
day,  prior  to  the  start  of  the  next  vignette.  Other  surveys  administered  during  MN  LOE  I  are 
discussed  in  Section  1.5. 


1.4  EVALUATION  STRATEGY 

1.4.1  COI#l 

“With  the  introduction  of  the  collaborative  tool,  did  the  Traditional  and 
Integrated  planning  processes  evolve  into  similar  processes  [from  the 
perspective  of  multinational  collaboration]  or  were  they  different? 

As  stated  previously,  three  COIs  were  specified  for  MN  LOE  I.  COI#l  addresses 
the  similarities  and  the  differences  of  the  two  planning  process  observed  during  MN  LOE  I. 
Given  the  complex  and  at  times  dynamic  nature  of  the  processes,  the  Traditional  and  Integrated 
processes  are  evaluated  on  the  basis  of  eight  common  characteristics:  Organizational  Structure, 
Use  of  Tool,  Schedule  of  Events,  Role  of  Lead  Nation,  Impact  of  Lead  Nation  on  Process, 
Effectiveness  of  Inter-Country  Interactions,  Effectiveness  of  Intra-Country  Interactions,  and 
Balance  in  Skills/Experience/Background  of  Team  Members.  The  comparison  of  the  two 
processes  is  based  on  these  eight  characteristics. 


12  “Autonomy  Portal-in-a-Box™  delivers  the  most  comprehensive  and  automated  Information  Portal  for  both  inside 
and  outside  the  firewall  in  one  easy  to  use  package.  Based  on  Autonomy ’s  unique  pattern-recognition  technology 
that  automatically  analyzes  information  based  on  its  content.  Autonomy  Portal-in-a-Box  ™is  the  only  portal 
solution  able  to  automate  the  most  critical  processes  including  categorization,  personalization,  hypertext  link 
management,  and  highly  personalized  information  delivery.”  rhttp://www. autonomy, com] 

13  Originally  the  length  of  the  vignettes  was  expected  to  range  from  6  to  12  hours.  After  the  start  of  the  LOE  the 
control  cell  decided  to  limit  the  vignettes  to  no  more  than  10  hours.  This  was  done  in  an  attempt  to  establish  an 
operational  tempo  for  the  LOE  that  would  keep  players  sufficiently  engaged. 
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1.4.2  COI  #2 

“Assuming  both  processes  are  equipped  with  the  same  collaboration 
capability,  does  the  quality  of  the  CO  As  produced  by  the  Traditional  and 
Integrated  planning  processes  differ  as  a  function  of  length  of planning 
window  and  type  of  operation?  Specifically,  does  one  process  produce  a 
better  product  (i.e.  Course  of  Action)  when  planning  time  is  short? 

COI#2  requires  a  comparative  assessment  of  the  Courses  of  Action  developed  by 
the  two  teams.  The  evaluation  of  the  COAs  is  based  on  three  Measures  of  Effectiveness 
(MOEs):  completeness,  accuracy  and  suitability. 

•  Suitability:  “Does  the  CO  A  support  the  Commander’s  intent  and  desired  end-state?  Is 
the  COA  acceptable  to  multinational  partners  and  the  host  country?” 

•  Completeness:  “Does  the  COA  lack  certain  relevant  and  critical  information?” 

•  Accuracy:  “Does  the  COA  include  errors  due  to  incorrect  information  or  assumptions?” 

The  evaluation  strategy  adopted  for  MN  LOE  I  was  to  have  all  evaluators,  a  panel 
of  senior  subject  matter  experts  (a.k.a.  gray  beards)  and  LOE  participants,  evaluate  the  COAs 
produced  by  the  two  teams.  The  evaluators  would  apply  the  three  MOEs  to  the  three  sections  of 
the  COA  (see  Table  1-1)  and  identify  any  and  all  deficiencies.  Factors  to  be  considered  when 
completing  the  evaluation  questionnaire  were  provided  as  guidance  (see  Figure  1-3),  but 
primarily  the  evaluators  were  expected  to  base  their  critiques  on  personal  operational  experience. 
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Table  1-1 


COA  Template 


COA  Sections 

COA  Sub-sections 

I:  CONOPS/Tasks 

Summary  of  Concept  of  Operations 

Anticipated  End-State 

Tasks  and  Supporting  Commander 

ROE  Considerations 

Potential  Adverse  Activity 

Information  sources  and  assumptions 

II:  Requirements 

Forces  Required;  Considerations 

Logistics;  Considerations 

Intelligence;  Considerations 

Other;  Considerations 

Infrastructure  Requirements/Shortfalls 

Anticipated  Host  Country  Support 

Information  sources  and  assumptions 

III.  Likelihood  of  Success 

Likelihood  COA  Will  Succeed 

Anticipated  Timeframe  for 

Completion 

Rationale:  Constraints 

Rationale:  Risks 

Information  sources  and  assumptions 
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MOEs 

(/) 

+-• 

c 

C  Actions 

c 

o 

o. 

P 

Completeness 

Accuracy 

Suitability 

Factors  to  consider  include: 

Omissions  of  critical  actions  required  to 
address  unique  operational  situation, 
critical  data  sources  not  considered, 
specification  of  chronology  of  events, 
etc. 

Factors  to  consider  include: 

Use  of  erroneous  data/assumptions  or 
incorrect  interpretation  of  information  in 
developing  action  plan 

Factors  to  conside  include: 

Support  of  Commander's  intent, 
support  of  desired  end  state, 
consistency  with  guidance/checklists 
provided  in  Joint  Doctrine  publications, 
etc. 

o 

o 

^  Requirements 

< 

o 

o 

Likelihood  of 

Success 

Factors  to  consider  include: 

Omissions  in  support  requirements, 
critical  data  sources  not  considered 

Factors  to  consider  include: 

Errors  in  determining  impact  of  COA  on 
Pacifica  mission,  Errors  in  assessment 
of  MNF  capabilities/resources,  Errors 
on  assessment  of  host  country 
capabilities/resources 

Factors  to  consider  include: 

Impact  on  ability  to  support  Pacifica 
operation 

Factors  to  consider  include: 

#  of  (known)  risks  not  considered, 
omissions  in  understanding  of  enemy 
vulnerabilities,  omissions  in 
understanding  ofMNF  vulnerabilities 

Factors  to  consider  include: 

Validity  of  assessment  of  host  country 
objectives  and  response 

Factors  to  consider  include: 

Consistency  with  relevant  ROEs  and 
doctrine;  MNF  support  for  the  COA; 
likelihood  of  COA  to  succeed;  Number 
of  constraints  impacting  success  of 
COA;  Flexibility  to  accommodate 
redirection 

Figure  1-3:  Factors  to  Consider  in  Applying  MOEs  to  COA 


Immediately  after  each  vignette  the  participants  were  asked  to  complete  a 
Vignette  Evaluation  Questionnaire  that  included  a  section  dedicated  to  the  evaluation  of  the 
COAs.  Each  reviewer  was  to  rate  the  three  sections  of  the  COA  as  totally  satisfactory  or  less 
than  totally  satisfactory  in  terms  of  completeness,  accuracy  and  suitability.  For  all  ratings  of 
“less  than  totally  satisfactory,”  the  reviewer  was  to  specify  and  weight  the  deficiencies. 

After  the  completion  of  MN  LOE  I  a  panel  of  multinational  senior  subject  matter 
experts  was  also  asked  to  evaluate  the  COAs  produced  by  the  two  planning  teams.  A  summary 
of  the  graybeard  evaluation  process  and  results  is  provided  in  Section  3. 

1.4.3  COI  #3 

“Can  collaborative  planning  be  conducted  successfully  in  a  distributed 

environment? 

COI  #3  explores  the  viability  of  conducting  planning  in  a  distributed  environment. 
The  assessment  of  COI  #3  is  based  on  a  top  level  critique  of  the  COAs  produced  by  the  planning 
teams  (i.e.,  “Were  the  teams  able  to  produce  reasonable  COAs?”),  the  observations  of  the  control 
cell,  and  participant  feedback  provided  via  the  “Team  Working  and  Planning  in  Coalition 
Distributed  Teams  Survey”  administered  on  the  last  day  of  the  LOE. 


1-9 


JWR-02-007 


MN  LOE I 


1.5  DATA  COLLECTION  STRATEGY 


Data  from  various  sources  were  required  to  adequately  evaluate  the  three  COIs. 
The  data  collection  strategy  implemented  for  MN  LOE  I  was  extensive  and  included  the  efforts 
of  observers  during  vignette  play  and  significant  data  archival  after  each  vignette.  The  data 
sources  are  detailed  below: 

•  Planning  Products:  Mission  Analysis  products14  and  COAs  generated  by  the  two  teams 
were  archived  after  every  vignette. 

•  Vignette  Evaluation  Questionnaires:  The  Vignette  Evaluation  Questionnaire  was 
comprised  of  three  sections:  CO  A  Evaluation,  Process  Evaluation,  and  LOE  Evaluation. 
The  participants  completed  this  questionnaire  after  each  vignette.  The  objective  of  the 
COA  Evaluation  section  was  to  collect  comments  pertaining  to  the  completeness, 
accuracy  and  suitability  of  the  COAs  developed  using  the  Traditional  and  Integrated 
processes.  The  Process  Evaluation  section  solicited  comments  regarding  strengths  and 
weaknesses  of  the  processes  as  well  as  recommended  improvements.  The  LOE 
Evaluation  section  solicited  comments  pertaining  to  LOE  tools  and  structure. 

•  Player  Profile  Surveys:  A  critical  aspect  of  post-LOE  analysis  was  the  review  of  the 
evaluation  questionnaires  completed  by  participants.  The  Player  Profile  Survey  provided 
the  analysis  team  with  insight  into  the  experiences  (operational,  training,  academic,  etc.) 
influencing  the  responses. 

•  Process  Description  Sheets  and  Control  Cell  Observations:  Throughout  the  LOE 
members  of  the  control  cell  observed  the  activities  of  the  planning  cells  at  each  of  the 
sites.  When  required,  they  interacted  with  the  players  to  gather  information  and  team 
leaders  were  routinely  interviewed.  Control  cell  members  also  monitored  Groove  spaces 
to  track  collaboration  activities.  Since  it  was  anticipated  that  the  Traditional  and 
Integrated  processes  would  evolve  over  the  course  of  the  experiment,  the  Process 
Description  sheet  was  developed  to  help  control  cell  staff  track  significant  changes. 
Control  cell  members  completed  this  form  at  the  end  of  every  vignette. 

•  Groove  Space  Tools  (Threaded  Discussion,  Text  Chat,  Messages):  The  threaded 
discussion  and  text  chat  areas  stored  in  the  various  shared  spaces  were  collected  to  aid  the 
analysis  team  in  understanding  the  thought  processes  involved  in  developing  the  COAs 
and  also  reconstruct  the  timelines  of  the  planning  processes. 

•  Team  Working  and  Planning  in  Coalition  Distributed  Teams  Survey:  This  survey  was 
designed  to  gain  insight  into  the  experiences  of  LOE  participants  working  as  part  of  a 
distributed  international  team.  It  included  questions  pertaining  to  interoperability  and 
distributed  team  working. 

•  Analysis  Workshop:  A  workshop  was  held  approximately  1 .5  months  after  MN  LOE  I 
during  which  analysts  from  the  participant  countries  were  able  to  review  findings  and 
share  insights.  The  workshop  is  discussed  further  in  Section  3. 


14  Unlike  the  COA,  the  Mission  Analysis  products  were  defined  by  the  teams  and  consequently  differed  somewhat 
between  the  Traditional  and  Integrated  teams. 
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Section  2 

SUMMMARY  OF  MN  LOE  I 

2.1  LOE  MANAGEMENT 


The  LOE  was  managed  during  execution  by  a  distributed  virtual  control  cell  that 
was  linked  via  the  CFBLNet.  The  main  component  of  the  control  cell  was  located  at  the  US 
Joint  Forces  Command  JBC  in  Suffolk,  Virginia.  The  control  cell  had  a  representative  at  each 
nation’s  point  of  participation.  The  control  cell  was  responsible  for  the  setup,  execution,  and 
data  collection  for  each  vignette.  The  control  cell  was  also  responsible  for  coordinating  with 
CFBFNet  operations  staff  to  resolve  network  difficulties.  Figure  2-1  depicts  the  distributed 
control  cell  for  the  LOE 


2.2  LEAD  ASSIGNMENTS  AND  VIGNETTES 


The  LOE  was  made  up  of  two  training  vignettes  followed  by  eight  vignettes. 
Each  vignette  exercised  two  teams,  Traditional  and  Integrated,  concurrently.  Each  team  was 
comprised  of  staff  from  each  participating  nation.  For  each  vignette,  each  team  had  a  lead  staff 
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designated  among  the  participating  nations.  If  a  national  staff  was  designated  as  lead,  that  nation 
provided  a  staff  of  five  individuals;  otherwise,  national  staffs  consisted  of  three  individuals. 

Each  traditional  national  staff  was  scheduled  to  lead  twice  during  the  LOE.  The  US  staff  led  the 
integrated  team  during  the  training  vignettes  and  the  vignettes  of  the  first  week,  since  the  US 
staff  was  most  familiar  with  the  Integrated  approach.  This  provided  the  other  national  staffs  an 
opportunity  to  observe  before  leading  the  Integrated  team.  Each  participating  nation  was 
scheduled  to  lead  the  Integrated  team  once  during  the  second  week.  The  rotation  of  lead  staffs 
was  scheduled  so  that  no  nation  had  the  lead  for  both  teams  concurrently,  and  therefore  would 
not  be  required  to  provide  more  than  eight  individuals  during  any  vignette.  Figure  2-2  depicts 
the  LOE  schedule  and  assignments. 


The  control  cell  used  the  same  Groove  peer-to-peer  collaboration  software  for 
coordination  as  did  the  LOE  participants  during  the  vignettes.  This  simplified  the  software 
configuration  and  permitted  the  control  cell  to  observe  participant  collaboration  during  the 
vignette  yet  gave  the  control  cell  a  private  control  space  for  coordination.  Prior  to  the  start  of  a 
vignette,  the  control  cell  would  create  all  the  shared  spaces  needed  for  the  vignette,  ensure  all 
background  and  scenario  infonnation  was  available  in  the  spaces,  and  verify  that  all  participants 
had  access  to  the  shared  spaces.  Each  national  Traditional  staff  had  a  shared  space  for  planning 
and  it  was  up  to  the  lead  nation  to  decide  how  collaboration  was  to  take  place  across  national 
staffs.  The  Integrated  team  had  a  main  shared  space  and  three  functional  shared  spaces  for 
planning,  intel,  and  logistics,  which  could  be  accessed  by  all  members  of  the  Integrated  team 
from  all  nations.  All  participants  had  access  to  a  scenario  space  that  provided  background 
information.  During  the  vignettes,  the  control  cell  distributed  scenario  events,  like  intelligence 
reports,  to  all  participants  on  both  teams  by  instant  text  messages  so  there  was  no  bias  towards 
any  team  organization. 
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During  the  conduct  of  a  vignette,  the  distributed  members  of  the  control  cell 
monitored  activity  at  their  location  and  used  the  control  shared-space  to  coordinate  as  necessary, 
usually  by  text  or  voice  chat.  The  CFBLNet  operations  staff  also  had  access  to  the  control 
shared  space  to  facilitate  coordination  when  network  problems  arose.  This  was  important  since 
the  Groove  software  requires  connectivity  to  synchronize  shared  spaces  in  each  Groove  client, 
the  collaboration  software  application  that  resides  on  the  computer  of  each  participant. 

The  control  cell  relied  heavily  on  observations  during  the  LOE  to  provide  the 
context  for  participant  actions  and  activity.  The  control  cell  members  at  the  overseas  locations 
coordinated  with  national  technical  staff  to  collect  observations  on  survey  forms  that  were  later 
integrated  into  the  LOE  analysis  effort.  The  following  organizations  provided  observers  and 
survey  materials  that  contributed  to  the  analysis: 

•  Defence  Science  &  Technology  Organisation  (DSTO)  -  Fern  Hill  Park,  Australia 

•  Canadian  Forces  Experimentation  Centre  (CFEC)  -  Ottawa,  Canada 

•  German  Joint  Operations  Command  -  Potsdam,  Germany 

•  Defence  Science  &  Technology  Laboratory  (DSTL)  -  Portsdown  West,  EfK. 

Figure  2-3  is  a  graphical  representation  of  the  Groove  shared  spaces  used  during 
each  LOE  vignette. 
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Figure  2-3:  LOE  Vignette  Collaboration  Spaces 
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2.3  SYSTEM  PERFORMANCE 


The  CFBLNet  permitted  the  MN  LOE  I  to  be  conducted  in  a  distributed  global 
manner,  allowing  nations  to  participate  from  their  own  facilities  in  their  country.  By  taking 
advantage  of  the  CFBLNet  and  the  available  national  facilities,  a  considerable  savings  in  travel 
costs  was  possible.  Having  a  point  of  participation  located  in  each  nation  also  facilitated  the 
availability  of  national  staff  to  participate  in  the  MN  LOE.  Conducting  the  first  LOE  distributed 
over  a  global  wide  area  network  did  raise  the  issue  of  risk.  A  big  risk  mitigating  factor  was  that 
CFBLNet  successfully  supported  the  Joint  Warrior  Interoperability  Demonstration  (JWID)  2001 
earlier  in  the  year  and  substantial  technical  support  was  available  in  each  participating  nation  and 
at  critical  nodes.  Critical  nodes  included  the  Defense  Information  Systems  Agency  (DISA) 
Advanced  Information  Technology  Services  Joint  Program  Office  (AITS-JPO)  and  NATO 
Command  Control  Communications  Agency  (NC3A).  NC3A  was  a  key  node  on  CFBLNet  for 
the  Gennan  Point  of  Participation  (POP)  in  Potsdam,  which  did  not  exist  during  JWID  2001. 

The  German  POP  at  Potsdam  was  linked  through  NC3A  to  the  CFBLNet  at  Royal  Air  Force 
(RAF)  Molesworth,  UK. 

Three  stress  tests  with  all  POPs  were  conducted  to  ensure  that  the  use  of  Groove 
over  CFBLNet  could  support  the  MN  LOE.  Of  particular  concern  was  the  German  POP,  which 
was  linked  to  the  CFBLNet  by  commercial  Integrated  Services  Digital  Network  (ISDN)  dial-up 
lines  through  NC3A.  Initially,  the  connection  to  Gennany  had  512  Kbps  of  bandwidth  from 
RAF  Molesworth  to  NC3A  and  256  Kbps  from  NC3A  to  Potsdam.  However  testing  showed 
1  Mbps  from  RAF  Molesworth  to  NC3A  and  512  Kbps  from  NC3A  to  Potsdam  were  needed  to 
reduce  latency  to  an  acceptable  level.  From  the  US,  CFBLNet  provided  5 12  Kbps  to  Australia, 
1.5  Mbps  to  Canada,  and  2  Mbps  to  the  UK. 

The  stress  tests  were  also  designed  to  determine  if  the  Groove  collaboration 
software  would  perfonn  as  advertised.  The  MN  LOE  planning  team  used  Groove  on  the  Internet 
as  the  planning  space  for  the  LOE  with  numerous  participants  from  all  participating  nations  with 
great  success.  Prior  to  MN  LOE  I,  Groove  had  not  been  tested  in  an  environment  where  up  to 
fifty  Groove  clients  would  be  synchronized  concurrently  over  a  network  with  single  paths  to  the 
POPs. 


A  Groove  enterprise  requires  a  relay  server  to  establish  the  presence  of  each 
Groove  client.  The  relay  server  provides  each  client  with  routing  information  to  deliver  data 
packets  known  as  “deltas”  from  one  client  to  another,  or  peer-to-peer  (P2P).  The  relay  server 
stores  deltas  that  cannot  be  immediately  delivered  from  one  client  to  another.  The  relay  server 
holds  the  deltas  for  a  period  of  time  then  delivers  the  deltas  to  the  receiving  client  when  it  is 
available.  Groove  Networks  provided  a  beta  version  of  their  Internet  relay  server  for  use  on  the 
CFBLNet  during  the  LOE.  The  beta  relay  server  was  limited  in  that  it  could  not  forward  deltas 
to  clients  who  did  not  receive  them  when  they  were  first  sent.  This  required  the  sending  client  to 
send  deltas  again  when  the  receiving  client  was  available. 

The  three  stress  tests  and  two  training  vignette  sessions  led  the  MN  LOE  planning 
team  to  believe  that  Groove  and  the  CFBLNet  as  configured  for  the  LOE  were  sufficient  to 
support  the  eight  vignettes  scheduled  for  the  two-week  period,  5-16  November  2001. 
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2.4  DATA  COLLECTED 


The  performance  of  the  collaboration  software  over  a  global  wide  area  network 
was  an  experiment  in  itself.  The  LOE  was  designed  to  examine  the  implementation  of  Joint 
Interactive  Planning  (JIP)  by  a  multinational  force  headquarters  planning  an  RDO  and  not  the 
technical  infrastructure  supporting  the  LOE.  However,  because  Groove  is  a  relatively  new 
collaboration  product  built  on  the  P2P  concept,  there  was  strong  interest  on  the  part  of  the 
participating  nations  in  gaining  further  insight  into  the  collaboration  tool  and  distributed  planning 
environment.  For  example,  DSTL  investigated  the  human  factors  associated  with  collaboration 
using  computer  software  over  a  wide  area  network.  DSTO  surveyed  the  LOE  participants  on  the 
use  of  Groove  as  a  collaboration  tool.  DSTL  and  DSTO  findings  will  be  published  separately. 

The  CFBLNet  was  generally  reliable  except  for  the  link  from  RAF  Molesworth  to 
Potsdam,  an  extension  to  the  CFBLNet  put  in  place  for  the  MN  LOE.  The  link  to  Potsdam  was 
made  up  of  separate  ISDN  lines,  each  with  64  Kbps  of  bandwidth.  For  example,  the  link  from 
NC3A  to  Potsdam  consisted  of  eight  ISDN  lines.  The  lines  were  running  Internet  Protocol  (IP) 
over  Asynchronous  Transfer  Mode  (ATM);  overhead  meant  the  usable  bandwidth  was  less  than 
64  Kbps.  There  were  continual  problems  with  lines  disconnecting  and  having  to  be  redialed.  On 
several  occasions  during  LOE  vignettes  the  link  to  Potsdam  went  completely  down  when 
commercial  providers  performed  maintenance,  usually  after  midnight  local  time. 

When  the  German  POP  would  go  down,  Groove  clients  at  all  other  POPs  would 
have  to  re-send  deltas  generated  while  the  Gennan  POP  was  down.  The  glut  of  deltas  severely 
affected  throughput,  especially  from  Australia  to  Germany.  There  was  no  way  to  purge  the 
backlog  of  deltas  in  Groove  other  than  to  “un-invite”  a  client  from  a  shared  space,  remove  the 
shared  space  from  the  client,  and  then  re-invite  the  client  to  the  same  space,  a  clumsy 
workaround  at  best.  Figure  2-4  shows  bandwidth  used  during  a  portion  of  the  LOE. 

The  graphs  in  Figure  2-4  represent  throughput  on  CFBLNet  in  and  out  of  the  US 
for  two  vignettes.  The  activity  on  the  left  of  the  upper  band  shows  the  loss  of  connectivity  to 
Germany  during  a  vignette  with  throughput  between  the  other  nations  well  within  available 
bandwidth.  On  the  right  is  the  network  activity  associated  with  the  subsequent  vignette.  Here 
network  activity  is  normal  with  the  exception  of  Australia,  where  Groove  clients  were  re-sending 
deltas  that  were  not  previously  delivered. 
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Figure  2-4:  CFBLNet  Performance 

The  use  of  a  Network  Time  Protocol  (NTP)  is  important  when  using  Groove  as  a 
collaboration  tool.  Groove  not  only  ensures  that  shared  spaces  are  synchronized  on  all  clients,  it 
delivers  the  deltas  in  the  order  created.  If  Groove  client  computers  clocks  are  not  synchronized, 
deltas  will  be  delivered  in  the  order  they  were  created.  For  example,  if  computer  A  shows  a 
clock  time  of  8:00  and  computer  B  shows  a  clock  time  of  8:05,  any  deltas  created  by  computer  A 
will  appear  in  the  shared  space  on  computer  B  five  minutes  before  they  were  created.  Groove 
uses  the  delta  timestamp  to  maintain  chronological  order.  This  ordering  is  most  notable  in  text 
chat  and  can  be  confusing,  especially  if  users  are  exchanging  questions  and  answers.  For  the 
LOE,  different  clock  synchronization  solutions  were  used.  German  and  UK  computers  were 
synchronized  to  an  Internet  site  providing  Greenwich  Mean  Time  (GMT).  Australian  computers 
were  synchronized  to  a  router  at  the  JBC  set  to  GMT.  The  latter  worked  well  until  the  JBC 
router  lost  power  and  reverted  to  a  time  that  set  Australian  computer  clocks  back  seven  hours. 
When  the  router  was  reset  back  to  GMT,  Australian  Groove  clients  began  re-sending  seven  hours 
of  deltas.  An  important  lesson  learned  is  that  computer  clock  synchronization  is  critical  in  a 
distributed  collaboration  environment. 

In  the  final  analysis,  when  CFBLNet  was  stable,  Groove  performed  well.  Voice 
chat,  the  feature  most  susceptible  to  bandwidth,  had  low  latency  even  between  Australia  and 
Germany,  the  path  with  the  least  bandwidth  and  the  most  “hops”  as  shown  in  the  Figure  2-5. 


Australia  United  States  United  Kingdom  NC3A  Germany 


Figure  2-5:  CFBLNet  Bandwidth  for  MN  LOE  I 
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Figure  2-6  depicts  the  overall  reliability  of  the  CFBLNet  during  the  LOE.  The 
only  recurring  problems  were  with  the  ISDN  dial-up  lines  between  RAF  Molesworth,  NC3A, 
and  Potsdam.  Other  problems  over  the  course  of  the  LOE  were  a  JBC  power  failure  that  took 
down  a  router,  ATM  multi-point  signaling  and  crypto  sync  problems  at  NC3A,  and  an 
intennittent  Open  Shortest  Path  First  (OSPF)  routing  problem  in  the  US.  AITS-JPO  and  NC3A 
technical  staff  worked  quickly  to  resolve  problems  when  they  arose.  The  link  to  Potsdam  was 
clearly  a  case  of  what  was  feasible  within  the  time  available.  The  use  of  leased  commercial  dial¬ 
up  lines  was  the  only  available  solution.  A  more  permanent  solution  is  needed  if  the  German 
POP  is  to  remain  in  Potsdam. 
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Figure  2-6:  MN  LOE  I  Network  Reliability 


MN  LOE  I  was  the  first  multinational  effort  of  the  USJFCOM  Joint  Futures  Lab. 

It  was  unique  since  it  was  a  focused  discovery  experiment  structured  to  examine  a  specific  aspect 
of  the  RDO  concept,  namely  distributed  planning  with  multinational  partners.  The  LOE  was  also 
a  pilot  event  in  that  it  attempted  a  global  distributed  experiment  linking  sites  that  had  never  been 
linked  using  a  relatively  new  collaboration  tool  and  approach.  MN  LOE  I  also  took  an 
innovative  approach  for  controlling  the  experiment  by  relying  on  a  distributed  virtual  control  cell 
in  lieu  of  the  more  traditional  centralized  approach.  Finally,  MN  LOE  I  served  as  a  venue  for 
focused  research  on  topics  of  national  interest.  Follow-on  LOEs  will  benefit  from  the 
management  insights  gained  during  MN  LOE  I. 
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Section  3 

ANALYSIS  RESULTS 

3.1  INTRODUCTION 


The  data  collected  during  MN  LOE  I  were  both  massive  and  diverse.  They 
consisted  of  control  cell  observations,  participant  responses  to  several  surveys,  and  various 
planning  materials  stored  in  the  collaboration  tool.  The  primary  objective  of  post-LOE  analysis 
activities  was  to  compile  and  correlate  the  diverse  data  sources  in  order  to  address  the  three  COIs 
specified  for  MN  LOE  I: 

COI  #1:  With  the  introduction  of  the  collaborative  tool,  did  the  Traditional  and 

Integrated  planning  processes  evolve  into  similar  processes  [from  the  perspective  of 
multinational  collaboration]  or  were  they  different? 

Given  the  complex  nature  of  the  planning  processes,  the  study  of  processes  addressed  a 
set  of  common  characteristics:  Organizational  Structure,  Use  of  Tool,  Schedule  of 
Events,  Role  of  Lead  Nation,  Impact  of  Lead  Nation  on  Process,  Effectiveness  of  Inter- 
Country  Interactions,  Effectiveness  of  Intra-Country  Interactions,  and  Balance  in  Skills/ 
Experience/Background  of  Team  Members.  These  characteristics  will  be  addressed  in 
the  discussion  of  COI  #1. 

COI  #2:  Assuming  both  processes  are  equipped  with  the  same  collaboration  capability, 
does  the  quality  of  the  CO  As  (completeness,  accuracy  and  suitability)  produced  by 
the  Traditional  and  Integrated  planning  processes  differ  as  a  function  of  length  of 
planning  window  and  type  of  operation  ? 

COI  #3:  Can  collaborative  planning  be  conducted  successfully  in  a  distributed 
environment? 

The  Johns  Hopkins  University  Applied  Physics  Laboratory  (JHU/APL)  analysis 
team  reviewed  the  raw  data  collected  during  the  LOE  and  generated  preliminary  findings  to 
support  evaluation  of  the  COIs.  Table  3-1  lists  the  data  sources  used  to  evaluate  each  of  the 
three  COIs.  Preliminary  findings  were  reviewed  with  analysts  from  J9,  SACLANT  (NATO), 
UK,  Germany,  and  Canada  during  the  MN  LOE  I  Analysis  Workshop  conducted  at  JHU/APL 
January  9-10,  2002.  Inputs  from  the  multinational  analysts  have  been  incorporated  into  this 
report.  In  particular,  the  analysts  attending  the  workshop  participated  in  voting  exercises 
developed  to  answer  the  questions  posed  in  COI  #1  and  COI  #3. 
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Table  3-1 

Data  Sources  Used  to  Evaluate  MN  LOE  I  COIs 


COI 

DATA  SOURCE 

Control  Cell  Observations 

Vignette  Evaluation 

Surveys 

Planning  Materials  (e.g., 
Text  Chat) 

Distributed  Team  Work 
Survey 

Graybeard  COA 

Evaluations 

COI  #  1:  With  the  introduction  of  the  collaborative  tool,  did 
the  Traditional  and  Integrated  planning  processes  evolve 
into  similar  processes  [from  the  perspective  of  multinational 
collaboration]  or  were  they  different? 

Organizational  Structure 

Use  of  Tool15 

Schedule  of  Events 

Role  of  Lead  Nation 

Impact  of  Lead  Nation  on  Process 

Effectiveness  of  Inter-Country  Interactions 

Effectiveness  of  Intra-Country  Interactions 

Balance  in  S/E/B  of  Team  Members 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

COI  #  2:  Assuming  both  processes  are  equipped  with  the 
same  collaboration  capability  (i.e.,  Groove),  does  the  quality 
( completeness ,  accuracy,  suitability )  of  the  COAs  produced 
by  the  Traditional  and  Integrated  planning  processes  differ 
as  a  function  of  length  of  planning  window  and  type  of 
operation 

X 

COI  #3:  Can  collaborative  planning  be  conducted 
successfully  in  a  distributed  environment? 

X 

X 

X 

X 

X 

15  The  Military  Systems  Experimentation  Branch  of  the  Australian  Defense  Science  and  Technology  Office  (DSTO) 
administered  a  detailed  technology  survey  that  focused  on  Groove.  Analysis  of  this  survey  will  be  done  by 
DSTO.  When  the  DSTO  results  are  available,  the  key  findings  from  this  study  will  be  correlated  with  tool-related 
findings  included  in  this  report. 
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The  three  COIs  are  discussed  in  detail  in  the  following  sections. 


3.2  SUMMARY  OF  KEY  FINDINGS 


The  key  findings  compiled  during  MN  LOE  I  analysis  are  summarized  below;  the 
rationale  and  supporting  discussion  are  detailed  in  the  sections  that  follow. 

3.2.1  COI#l 

“With  the  introduction  of  the  collaborative  tool,  did  the  Traditional  and 

In  tegrated  planning  processes  evolve  into  similar  processes  [from  the 

perspective  of  multinational  collaboration]  or  were  they  different?  ” 

The  Traditional  team  took  advantage  of  the  collaboration  tool  and  the  Traditional 
process  followed  in  MN  LOE  I  was  more  integrated  in  nature  than  the  conventional  ‘stove- 
piped’  approach  typically  associated  with  the  Lead  Nation  concept.  Further  investigation  is 
required  to  assess  whether,  given  the  availability  of  a  robust  collaboration  environment  and  a 
commonly  defined  process,  the  Traditional  process  might  naturally  evolve  into  something  similar 
to  the  Integrated  process. 

The  Traditional  and  Integrated  processes  followed  in  MN  LOE  I  demonstrated 
significant  similarities  and  differences,  and  consequently  a  definitive  comparison  of  the  two 
processes  proved  to  be  challenging.  The  processes  did  exhibit  noteworthy  differences  in  the 
areas  of  leadership  and  situational  awareness.  Issues  related  to  leadership  have  potential 
implications  for  doctrine,  procedures  and  training.  The  infrastructure  implications  related  to 
maintaining  continuous  situational  awareness  of  team  activity  merits  further  study.  System  and 
network  performance  data  collected  during  MN  LOE  I  should  provide  very  useful  insignts. 

By  design,  planning  during  MN  LOE  I  was  primarily  conducted  at  the  operational 
level.  It  would  be  of  interest  to  examine  the  adaptation  of  the  Integrated  process  to  the  tactical 
level  (i.e.,  more  detailed,  real-time  planning). 

Key  findings  for  each  of  the  eight  process  characteristics  are  provided  below: 
Organizational  Structure 

Both  planning  teams  used  the  All  Hands  model  for  review  activity. 

During  the  first  week  of  the  LOE  the  Traditional  team  primarily  used  the  National 
Cell  approach  for  development  and  completion.  During  the  second  week  the  team  experimented 
with  the  Functional  and  All  Hands  models  for  these  planning  activities.  Throughout  the  LOE  the 
multinational  members  of  the  Traditional  team  interacted  during  all  planning  activities,  largely 
due  to  the  fact  that  they  elected  to  share  a  common  Groove  space. 
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Overall,  the  Integrated  team  followed  the  strategy  laid  out  in  the  LOE  training 
materials,  which  made  extensive  use  of  the  Functional  Cell  and  All  Hands  models.  The 
consistency  of  the  Integrated  approach  was  due  in  large  part  to  fact  that  the  United  States  was 
assigned  the  lead  for  four  of  the  seven  vignettes.  There  appeared  to  be  a  greater  emphasis  on 
following  a  constant  approach  vice  experimenting  and  refining  the  Standard  Operating 
Procedure. 

Use  of  Tool 


Both  planning  teams  elected  to  use  a  common  Groove  space  for  core  planning 
activities.  The  Traditional  cells,  however,  also  used  physical  whiteboards  whereas  the  Integrated 
team  conducted  all  work  in  the  Groove  shared  spaces. 

Both  planning  teams  used  the  full  suite  of  Groove  tools,  though  they  did  exhibit 
differences  in  how  they  used  the  tools.  The  most  significant  difference  between  the  two  teams 
was  in  their  use  of  audio,  with  the  Integrated  team  demonstrating  a  greater  use  of  audio.  Also, 
control  cell  observers  noted  that  the  Traditional  teams  experimented  more  with  Groove  tools 
than  did  the  Integrated  team.  The  Integrated  team  appeared  to  adhere  to  the  Groove  business 
rules  promulgated  at  the  beginning  of  the  LOE  in  order  to  replicate  the  same  process  for  each 
vignette. 


For  a  variety  of  reasons  neither  team  relied  heavily  on  BrainEKP  and  Autonomy. 
Both  teams  expressed  significant  satisfaction  with  Groove  and  an  interest  in  using  the 
collaboration  tool  in  a  real  operation. 

Schedule  of  Events 

The  Traditional  and  Integrated  planning  teams  followed  a  similar  schedule  of 
events  during  their  respective  planning  cycles.  The  Integrated  team  did  spend  more  time  on  MA 
development  (on  average),  while  the  Traditional  team  spent  more  time  on  COA  development  (on 
average).  Given  that  time  was  not  specified  as  a  measure  of  effectiveness  (MOE)  for  the 
planning  processes,  little  more  should  be  inferred  into  the  above  observations  other  than  the 
Integrated  process  appears  to  place  a  greater  emphasis  on  MA  development  as  compared  to  the 
Traditional  process. 

Role  of  Lead  Nation 

The  processes  followed  by  the  teams  during  the  LOE  appeared  to  embody 
different  perspectives  regarding  the  role  of  the  lead  country.  In  the  Traditional  process  “lead 
country”  was  synonymous  with  “leader;”  and  throughout  the  LOE,  team  members  examined  and 
reflected  on  the  topic  of  leadership.  By  comparison,  in  the  Integrated  process  the  lead  country 
operated  more  as  a  manager  than  a  traditional  Lead  Nation.  There  also  was  an  implicit 
assumption  that  any  planning  activity  conducted  by  the  lead  country  should  be  visible  to  the 
entire  team. 
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Impact  of  Lead  Nation  on  Process 

Differences  in  training  and  lead  assignments  resulted  in  the  Integrated  team 
following  a  process  that  was  notably  more  stable  than  that  of  their  Traditional  counterparts.  The 
impact  of  doctrinal  and  cultural  preferences  was  more  pronounced  in  the  Traditional  team; 
doctrinal  differences  were  often  cited  in  the  feedback  provided  by  the  Traditional  team. 

Effectiveness  of  Inter-Country  Interaction 

During  each  vignette  both  teams  successfully  generated  a  COA,  sometimes  in  the 
face  of  significant  network  problems.  The  Integrated  team,  more  than  the  Traditional  team, 
indicated  that  technical  problems  at  times  degraded  the  effectiveness  of  their  process,  primarily 
because  maintaining  situational  awareness  of  team  activities  is  critical  to  the  Integrated  process. 
By  contrast,  some  members  of  the  Traditional  team  indicated  that  at  times  there  was  too  much 
data  exchange  among  team  members. 

The  teams  did  not  provide  a  strong  indication  whether  the  distributed  environment 
made  it  more  difficult  to  develop  and  maintain  situational  awareness  or  anticipate  the  needs  of 
fellow  members.  Likewise,  few  participants  indicated  that  it  was  more  difficult  to  “reach 
consensus  over  decisions  in  a  distributed  team”. 

Effectiveness  of  Intra-Country  Interaction 

For  both  processes  members  of  the  national  cell  communicated  with  each  other 
primarily  using  face-to-face  communications  vice  the  collaboration  tool. 

Balance  of  Skil Is/Experience/Background  of  Team  Members 

Both  planning  teams  indicated  they  lacked  logistics  expertise.  The  US  Integrated 
team  was  unique  in  that  some  members  had  worked  together  for  almost  a  year  while  most  of  the 
other  cells  were  working  together  for  the  first  time  during  MN  LOE  I. 

3.2.2  COI  #2 

“ Assuming  both  processes  are  equipped  with  the  same  collaboration 
capability,  does  the  quality  of  the  COAs  (completeness,  accuracy  and 
suitability)  produced  by  the  Traditional  and  Integrated  planning  processes 
differ  as  a  function  of  length  of planning  window  and  type  of  operation? 

Overall,  the  graybeards  marginally  preferred  the  COAs  produced  by  the 
Integrated  team,  citing  completeness  as  a  critical  factor.  Given  the  small  size  of  the  graybeard 
panel  (five),  statistical  significance  cannot  be  assigned  to  most  of  the  individual  vignette  scores. 
Two  exceptions  are  Vignettes  E  and  H,  where  a  significant  majority  of  the  experts  selected  the 
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COAs  produced  by  the  Integrated  team. 

The  analysis  of  the  graybeard  evaluations  did  not  produce  a  definitive  explanation 
of  the  factors(s)  causing  the  graybeards  to  prefer  the  Integrated  COAs.  The  examination  of  the 
types  of  operations  and  lengths  of  planning  window  represented  in  the  seven  vignettes  did  not 
suggest  an  overall  trend  that  would  readily  explain  the  graybeard  selections.  A  workshop 
participant  suggested  that  the  “Integrated  team  was  better  at  coming  up  with  a  COA  given  the 
restricted  (low  intensity)  operations.”  This  is  an  interesting  observation  that  requires  further 
study. 


Another  factor  that  merits  further  examination  is  the  fact  that  the  Integrated  team 
followed  a  process  for  which  they  recently  received  training  and  that  was  played  consistently 
throughout  the  experiment.  Overall,  this  may  have  allowed  the  Integrated  team  to  better  retain 
corporate  knowledge  and  leverage  off  previous  planning  efforts. 

3.2.3  COI  #3 

“Can  collaborative  planning  be  conducted  successfully  in  a  distributed 
environment?  ” 

MN  LOE  I  did  demonstrate  that  collaborative  planning  can  be  conducted  in  a 
distributed  environment.  Even  with  this  positive  result,  the  potential  adverse  impact  of  poor 
system  and  network  performance  on  the  planning  processes  should  not  be  ignored  nor 
minimized.  Additional  study  is  required  to  specify  the  system  requirements  associated  with 
supporting  distributed  planning,  especially  at  the  level  represented  in  the  Integrated  process.  By 
design  the  exchange  of  classified  data  was  not  addressed  in  MN  LOE  I,  but  will  need  to  be 
explored  in  future  LOEs. 


3.3  DETAILED  DISCUSSION  OF  COI  #1 

“With  the  introduction  of  the  collaborative  tool,  did  the  Traditional  and 
Integrated  planning  processes  evolve  into  similar  processes  [from  the 
perspective  of  multinational  collaboration]  or  were  they  different?  ” 

3.3.1  ORGANIZATIONAL  STRUCTURE 

For  each  of  the  seven  vignettes  played  during  LOE  I,  the  two  planning  teams 
established  an  organizational  structure  that  determined  how  participants  operated  during  MA  and 
COA  development,  review,  and  completion.  The  country  designated  as  the  lead  for  a  given 
vignette  detennined  the  organizational  structure.  For  this  reason,  the  structures  varied  somewhat 
vignette  to  vignette.  Overall,  three  basic  models  for  planning  interactions  were  observed  during 
the  LOE: 
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•  National  Cell:  When  a  given  planning  activity  (i.e.,  development,  review,  or 
completion)  followed  the  national  cell  approach,  the  initial  effort  for  the  activity  was 
conducted  within  the  country  cell  before  the  planning  product  was  disseminated  (i.e., 
made  “visible”)  to  the  coalition  partners.  National  cells  operated  either  as  a  contributing 
country  or  the  lead. 

•  Functional  Cell:  When  a  given  planning  activity  followed  the  functional  cell  approach, 
cells  comprised  of  multinational  planners  were  established  to  address  specific  functional 
areas  such  as  plans,  intelligence  and  logistics.16  In  some  cases  a  multinational  “Tiger 
Team”  was  assigned  the  responsibility  for  producing  an  initial  product.  In  another 
variation  on  the  functional  cell  concept,  countries  were  assigned  the  role  of  a  Component 
Commander  (i.e.,  Air,  Land,  Maritime,  and  Special  Operations  Forces). 

•  All  Hands:  In  the  case  of  “all  hands”  interaction,  the  entire  team  interacted  during  a 
planning  activity,  almost  as  independent  agents.  In  almost  all  cases,  both  teams  adopted 
the  “all  hands”  approach  for  MA  and  COA  review. 

To  characterize  how  the  organizational  strategies  selected  by  the  two  teams  varied 
over  the  course  of  the  LOE,  a  pictorial  technique  that  visually  captures  differences  and  trends 
was  developed.  This  technique  is  demonstrated  in  the  pre-LOE  characterizations  of  the 
Traditional  and  Integrated  planning  organizational  structures  depicted  in  Figure  3-1.  The 
organizational  structures  used  during  each  planning  phase  (MA  development,  review,  and 
completion;  COA  development,  review,  and  completion)  are  depicted.  Horizontal  bars 
correspond  to  structures  aligned  with  the  National  Cell  model,  whereas  vertical  bars  represent 
structures  aligned  with  the  Functional  Cell  models.  The  diagonal  bar  represents  the  All  Hands 
model.  The  organizational  structure  used  by  a  team  during  a  given  planning  phase  is  highlighted 
in  yellow. 
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Figure  3-1:  Pre-LOE  Characterizations  of  Traditional  and  Integrated  Planning  Team 

Organizational  Structures 


16  Logistics  was  identified  as  a  critical  functional  area.  However,  given  LOE  staffing  choices,  the  logistics  analysis 
performed  during  the  LOE  focused  almost  exclusively  on  force  requirements.  Both  the  Traditional  and 
Integrated  teams  indicated  that  they  lacked  personnel  with  a  strong  logistics  background. 
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Based  on  the  Multinational  Interoperability  Working  Group  (MIWG)  white  paper 
on  the  Lead  Nation  concept,  it  was  envisioned  that  the  Traditional  planning  team  would  use  the 
“National  Cell”  approach  almost  exclusively.  Training  materials  provided  by  the  Integrated 
team  prior  to  the  LOE  indicated  that  their  strategy  would  rely  heavily  on  the  Functional  Cell  and 
All  Hands  models.  The  organizational  structures  observed  during  the  seven  vignettes  of  MN 
LOE  I  are  documented  in  Appendix  B. 

As  previously  noted,  both  teams  used  the  All  Hands  approach  for  almost  all 
review  activity.  Also,  the  structures  of  both  teams  varied  over  the  course  of  the  LOE  depending 
on  the  direction  provided  by  the  “lead”  country. 

Key  Findings 


Both  planning  teams  used  the  All  Hands  model  for  review  activity. 

During  the  first  week  of  the  LOE  the  Traditional  team  primarily  used  the  National 
Cell  approach  for  development  and  completion.  During  the  second  week  the  team  experimented 
with  the  Functional  and  All  Hands  models  for  these  planning  activities.  Throughout  the  LOE  the 
multinational  members  of  the  Traditional  team  interacted  during  all  planning  activities,  largely 
due  to  the  fact  that  they  elected  to  share  a  common  Groove  space. 

Overall,  the  Integrated  team  followed  the  strategy  laid  out  in  the  LOE  training 
materials,  which  made  extensive  use  of  the  Functional  Cell  and  All  Hands  models.  The 
consistency  of  the  Integrated  approach  was  due  in  large  part  to  the  fact  that  the  US  was  assigned 
the  lead  for  four  of  the  seven  vignettes.  There  appeared  to  be  a  greater  emphasis  on  following  a 
constant  approach  vice  experimenting  and  refining  the  Standard  Operating  Procedure. 


3.3.2  USE  OF  TOOL 

Both  teams  conducted  planning  using  Groove.  A  summary  of  the  usage  levels  (H=high, 
M=medium,  L=low/none)  of  the  different  tools  available  via  Groove  is  provided  in  Figure  3-2. 
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1. 


2. 
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Both  teams  used  common  spaces  almost  exclusively  (vice  woiking  in  tlieii  own  National  spaces), 
and  so  both  teams  had  the  ca/Ktbility  to  piovide  input  riming  MA  and  COA  development  and  leview. 
and  maintain  situational  awaieness  of  othei  commies'  inputs.  The  actual  level  of  collaboration 
was  deteimined  by  how  the  lead  nation  elected  to  implement  the  planning  piocess  dining  a  given 
vignette. 

Difficult  to  assess;  messages  only  stoied  on  individual  woikstations. 


Figure  3-2:  Use  of  Groove  for  Coalition  Team  Interaction 


Prior  to  the  LOE  it  was  expected  that  the  Traditional  process  would  utilize  private 
(i.e.,  country)  shared  spaces  for  national  cell  planning  whereas  the  Integrated  team  would  operate 
primarily  from  a  common  Integrated  Planning  space.  What  occurred  during  the  LOE  is  that  from 
the  outset  both  teams  elected  to  use  a  common  Groove  space  for  core  planning  work.17  In  the 
case  of  the  Traditional  team  the  decision  to  use  a  single  space  was  largely  a  response  to 
unreliable  network  performance.  Control  cell  observers  noted,  however,  that  while  the  national 
cells  did  not  use  their  private  Groove  spaces  each  Traditional  cell  did  utilize  a  physical 
whiteboard  that  served  as  a  virtual  planning  space  for  the  national  cell.  By  contrast  the 
Integrated  team  did  not  use  white  boards  and  almost  all  work  was  conducted  in  the  shared 
spaces,  thus  allowing  all  team  members  to  have  up-to-date  situational  awareness  of  planning 
activities.  From  the  perspective  of  observers,  control  cell  staff  found  the  single  space  strategy  of 
the  Traditional  team  easier  to  track  than  the  multiple  space  approach  adopted  by  the  Integrated 
team.  Concern  was  raised  about  the  possible  implications  for  hand-off  when  planning  personnel 
or  ‘lead’  country  are  re-assigned.  Feedback  provided  by  members  of  the  Integrated  team, 
however,  indicates  that  participants  actively  involved  in  the  process  did  not  appear  to  have 
problems  tracking  planning  activities. 


17  Functional  cells  established  within  the  Integrated  team  did  use  special  purpose  shared  spaces  for  Plans,  Intel  and 
Logistics. 
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The  Traditional  and  Integrated  teams  used  the  Agenda,  Messaging,  Text  Chat  and 
Notepad  tools  at  about  the  same  level.  The  two  planning  teams  differed  significantly,  however, 
in  their  use  of  audio  during  planning.  The  Integrated  team  considered  audio  to  be  critical  to  their 

is* 

planning  process  and  used  it  throughout  the  LOE.  The  Traditional  team  decided  early  on  to 
minimize  their  use  of  audio,  primarily  because  of  connectivity  problems  but  also  because  they 
felt  text  chat  minimized  confusion  among  multinational  partners.  Team  views  regarding  the 
impact  of  audio  on  their  planning  process  are  summarized  in  Figure  3-3. 19 


Lack  of  continuous  audio  was  a  significant  detriment  to  collaboration. 


%  Agree  %  %  Disagree  %  No 

Undecided  Opinion 


Figure  3-3:  Vignette  Evaluation  Question  1:  “Lack  of  Continuous  Audio 
Was  a  Significant  Detriment  to  Collaboration.” 


The  teams  also  differed  in  their  use  of  the  Threaded  Discussion  and  Whiteboard 
tools,  with  the  Traditional  teams  demonstrating  greater  and  more  optimal  use  of  these  tools. 
Both  teams  used  the  Threaded  Discussion  tool  to  manage  Requests  for  Information  (RFIs),  but 
the  Traditional  team  also  used  this  tool  to  effectively  share  products  and  conduct  reviews. 
Control  cell  observers  noted  that  the  Traditional  teams  experimented  more  with  Groove  tools 
than  did  the  Integrated  team  over  the  course  of  the  FOE  resulting  in  greater  efficiency  in  their 
use  of  certain  tools.  A  factor  to  be  considered,  however,  is  that  the  Integrated  team  attempted  to 
replicate  the  same  process  during  each  vignette  and  therefore  may  have  elected  to  follow  the 
Groove  business  rules  promulgated  at  the  beginning  of  the  LOE. 


18  Though  the  majority  of  the  Integrated  team  held  this  view,  there  were  some  members  who  disagreed.  Country 
responses  to  this  question  and  participant  comments  regarding  the  use  of  audio  are  provided  in  Appendix  C. 

19  These  data  were  collected  from  the  Vignette  Evaluation  Questionnaires  completed  after  each  vignette.  Figure  2-4 
reflects  summary  totals  for  the  entire  LOE.  Breakdowns  by  week  and  country  are  provided  in  Appendix  C. 
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As  noted  earlier,  the  planning  teams  were  also  equipped  with  BrainEKP  and 
Autonomy  to  support  open  source  information  (OSI)  research  for  intelligence  purposes.  In  lieu 
of  a  true  Operational  Net  Assessment  for  the  geographic  areas  specified  in  the  vignettes, 
BrainEKP  was  intended  to  provide  limited  Intelligence  Preparation  of  the  Battlespace  (IPB). 
The  feedback  collected  from  the  participants  suggests  that  these  tools  were  not  used  very  much 
during  the  LOE.  Survey  statements  postulating  that  BrainEKP/ Autonomy  “supported  the 
planning  process  in  a  satisfactory  manner”  yielded  a  significant  number  of  “Undecided”  and 
“Not  Applicable”  responses,  indicating  that  many  participants  had  not  used  the  tools  enough  to 
be  able  to  agree  or  disagree  with  the  survey  statements.20  This  conclusion  is  confirmed  by  the 
observations  of  control  cell  staff  stationed  at  the  national  sites.  Lack  of  fonnal  training  prior  to 
the  LOE  and  unavailability  of  the  tools  due  to  network  problems  were  cited  as  potential  reasons 
why  the  participants  did  not  use  the  tools  more.  Control  cell  observers  also  noted  that  early  on 
some  participants  found  the  URLs  loaded  into  the  tools  to  be  irrelevant  to  the  problem  at  hand, 
and  therefore  elected  to  henceforth  ignore  the  tools  and  query  the  www  directly. 

Key  Findings 


In  summary,  both  planning  teams  elected  to  use  a  common  Groove  spaces  for 
core  planning  activities.  The  Traditional  cells,  however,  also  used  physical  whiteboards  whereas 
the  Integrated  team  conducted  all  work  in  the  Groove  shared  spaces. 

Both  planning  teams  used  the  full  suite  of  Groove  of  tools,  though  they  did 
exhibit  differences  in  how  they  used  the  tools.  The  most  significant  difference  between  the  two 
teams  was  in  their  use  of  audio,  with  the  Integrated  team  demonstrating  greater  use  of  audio. 
Also,  control  cell  observers  noted  that  the  Traditional  teams  experimented  more  with  Groove 
tools  than  did  the  Integrated  team.  The  Integrated  team  appeared  to  adhere  to  the  Groove 
business  rules  promulgated  at  the  beginning  of  the  LOE  in  order  to  replicate  the  same  process 
each  vignette 


For  a  variety  of  reasons  neither  team  relied  heavily  on  BrainEKP  and  Autonomy. 

Both  teams  expressed  significant  satisfaction  with  Groove  and  an  interest  in  using  the 

21 

collaboration  tool  in  a  real  operation.  (See  Figure  3-4.) 


20  Tool  evaluation  data  compiled  from  the  Vignette  Evaluation  Questionnaires  is  provided  in  Appendix  C. 

21  These  data  was  collected  from  the  Vignette  Evaluation  Questionnaires  completed  after  each  vignette.  Additional 
details  pertaining  to  this  data  are  provided  in  Appendix  C. 
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Groove  supported  the  planning  process  in  a  satisfactory  manner. 


|  I  would  like  to  use  Groove  in  a  real  operation. 
Traditional  Team 

2% 


Integrated  Team 


12%  2% 


57% 


■  Agree 

■  Undecided 

□  Disagree 

□  N/A 


Figure  3-4:  Groove  Evaluation  Questions  -  Vignette  Evaluation  Questionnaire 


3.3.3  SCHEDULE  OF  EVENTS 

Both  planning  teams  followed  a  planning  cycle  that  included  development, 
review,  and  completion  of  MA  products  and  a  COA.  Table  3-2  details  the  start  and  end  times  of 
these  planning  activities  for  each  of  the  seven  vignettes.  The  two  teams  followed  basically  the 
same  planning  agenda,  though  there  were  some  differences  in  the  respective  planning  timelines, 
as  summarized  in  Figure  3-5. 


3-12 


JWR-02-007 


MNLOEI 


Table  3-2 


Planning  Timelines 


Vignette  ID 

(Start  Time,  End  Time) 
DECRIPTION 
Designated  Timeline 

Timelines 
(All  Times  in  Zulu) 

Traditional  Process 

Integrated  Process 

Mission  Analysis 

Course  of  Action 

Mission  Analysis 

Course  of  Action 

C  (1200Z,  1800Z) 

TYPHOON  VIETNAM 
MA:1500,  COA:1800 

UK 

1204  Started 

1332  Review 
started 

1423  Completed 

UK 

1425  Started 

1700  Review 
started 

1750  Completed 

US 

1200  Started 

1445  Review 
started 

1522  Completed 

UK 

1528  Started 

1732  Review 
started 

1745  Completed 

E  (1200Z,  1800Z) 
PIRACY 

MA:1500,  COA:1800 

GE 

1205  Started 

1340  Review 
started 

1411  Completed 

GE 

1418  Started 

1613  Review 
started 

1655  Completed 

US 

1200  Started 

1340  Review 
started 

1449  Completed 

AUS 

1510  Started 

1708  Review 
started 

1806Z  Completed 

F  (1000Z,  2000Z) 

HOSTAGE 

CRUISE  LINER 

MA:  1500Z,  COA: 

2000Z 

UK 

1000  Started 

1127  Review 
started 

1245  Completed 

UK 

1236  Started 

1701  Review 
started 

1913  Completed 

US 

1000  Started 

1340  Review 
started 

1500  Completed 

US 

1508  Started 

1602  Review 
started 

1930  Completed 

G  (2200Z,  0400Z) 

NEO-INDONESIA 

MA:  0130Z,  COA:  0400 

GE 

2200  Started 

0009  Review 
started 

0105  Completed 

GE 

-0105  Started 

0253  Review 
started 

0355  Completed 

AUS 

2200  Started 

2340  Review 
started 

0130  Completed 

AUS 

0153  Started 

0330  Review 
started 

0414  Completed 

H  (2200Z,  0400Z)1 

PROTECTION  OF 
SHIPPING-  PIRACY 

MA:  0200Z,  COA: 

0600Z 

US 

2212  Started 

2342  Review 
started 

0033  Completed 

US 

0039  Started 

0337  Review 
started 

0445  Completed 

GE 

2200  Started 

2330  Draft/Review 
0109Completed 

UK 

0130  Started 

0334  Draft/Review 
0512  Completed 

I  (2300Z,  0700Z)2 

DISASTER  RELIEF 

PNG 

MA:0300,  COA:0700 

US 

2307  Started 

0100  Review 
started 

0220  Completed 

US 

0237  Started 

0504  Review 
started 

0538  Completed 

UK 

2300  Started 

0115  Draft/Review 
0153Completed 

UK 

0214  Started 

0315  Review 

Started 

0512  Completed 

J  (0230Z,  0800Z)3 

PIRACY:  MV  WEN 

ZHO 

MA:  0500Z,  COA: 

0800Z 

AU 

0230  Started 

0324  Review 
started 

0410  Completed 

AU 

0357  Started 

0523  Review 
started 

0652  Completed 

US 

0230  Started 

0340  Review 
started 

0520  Completed 

AUS 

0525  Started 

0628  Review 
started 

0737  Completed 

1  Vignette  H  -  Lost  Germany  approximately  01 14Z.  UK  took  the  lead  at  0141Z. 

2  Vignette  I-  Germany  lost  connectivity  ~  0354Z. 

3  Vignette  J-  Germany  lost  connectivity  -  all  night. 
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Traditional  Process  Planning  Time  Distributions 
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■  Unused  Time 

□  Review  CO  A 

□  Develop  COA 

■  Review  MA 

■  Develop  MA 


Vignette  (Total  Allowed  Time) 
(*  Means  Added  Time) 


Integrated  Process  Planning  Time  Distributions 
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■  Unused  Time 

□  Review  COA 

□  Develop  COA 

■  Review  MA 

■  Develop  MA 


Figure  3-5:  Planning  Time  Distributions  for  Traditional 
and  Integrated  Teams 


In  Figure  3-5,  the  time  for  “Develop  MA”  represents  the  time  from  the  start  of  the 
vignette  to  the  indicated  completion  of  the  draft  MA.  The  “Review  MA’  represents  the  time  from 
the  completion  of  that  draft  MA  to  the  completion  of  its  review.  Since  typically  only  a  few  minutes 
lapsed  before  the  development  of  the  COA  was  begun,  the  time  period  for  “Develop  COA”  is 
measured  from  the  completion  of  the  MA  Review  to  the  completion  of  the  draft  COA.  Similarly, 
“COA  Review”  is  measured  from  this  point  to  the  point  that  the  COA  is  finished  and  sent  to 
SACAPAC.  In  several  cases  for  the  Integrated  team  (specifically  Vignettes  F  and  I),  it  was  difficult 
to  ascertain  exactly  when  the  COA  draft  was  finished  and  its  review  was  started.  In  particular,  in 
Vignette  F,  it  appears  that  pieces  of  the  COA  were  finished  and  under  review  while  other  portions  of 
the  COA  were  still  in  development.  The  mid-point  of  that  process  was  used  as  the  dividing  line  for 
this  figure.  In  some  cases,  the  COA  was  finished  and  delivered  to  SACAPAC  before  the  end  of  the 
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planned  time  for  the  vignette.  This  excess,  unused  time,  is  shown  as  red.  In  three  cases,  Vignettes 
E,  G,  and  H,  the  Integrated  Team  took  slightly  longer  than  the  original  time  planned  for  the  vignette. 
This  time  overrun  is  considered  an  informal  extension  of  the  vignette  duration  and  shown  here  as 
“added  time,”  indicated  by  an  asterisk. 

Before  considering  findings  based  on  the  time  distributions  depicted  in 
Figure  3-5,  it  is  prudent  to  recognize  extenuating  factors  that  may  have  impacted  the  realized 
planning  time  distributions. 

•  “Dead  Time”:  Unused  time,  as  defined  above,  does  not  include  the  periods  when  the 
network  was  down  for  at  least  one  of  the  countries.  These  “dead”  periods  are  included  in 
the  planning  time  distributions.  As  noted,  during  some  vignettes  Germany  had  minimal 
or  no  connectivity.  Also,  control  cell  observers  noted  that  at  times  participants  in 
contributing  countries  were  relatively  inactive  while  awaiting  guidance  from  the  lead 
country. 

•  Similar  Vignettes:  Control  cell  staff  observed  that  because  some  vignettes  posed  similar 
situations,  there  were  cases  when  participants  leveraged  off  work  that  had  been 
performed  during  an  earlier  vignette  (in  particular  during  the  MA  phase).  This 
observation  applies  primarily  to  the  Traditional  team. 

•  Lead  Hand-Off:  Because  of  network  problems,  there  were  occasions  when  the  lead 
assignment  had  to  be  transferred  to  another  country. 

Examining  the  planning  time  distributions  for  MA  and  COA  efforts  reveals  that 
there  are  some  differences  between  the  teams  in  the  times  allocated  to  tasks  but  no  clear  trends 
are  noted  through  the  experiment  for  changes  in  those  times.  It  is  the  case  that  both  teams  spent 
on  average  58%  of  the  planning  time  on  development.  On  average,  the  Integrated  team  spent 
more  time  on  MA  development  than  did  the  Traditional  team  (30%  vice  23%)  while  the  converse 
is  true  for  COA  development  (Traditional  35%;  Integrated  28%).  Discussions  with  US 
participants  revealed  that  spending  more  time  on  MA  was  viewed  as  contributing  to  a  better 
COA  product  and  actually  an  integral  part  of  the  Integrated  strategy.  Typically,  a  preset  time 
interval  was  allocated  for  MA  development  and  in  most  cases  the  entire  interval  was  used.  On 
the  other  hand,  the  operational  tempo  of  MA  development  for  the  Traditional  team  was  driven 
more  by  the  expertise  and  initiative  of  the  contributing  countries. 

Key  Findings 


The  Traditional  and  Integrated  planning  teams  followed  a  similar  schedule  of 
events  during  their  respective  planning  cycles.  The  Integrated  team  did  spend  more  time  on  MA 
development  (on  average),  while  the  Traditional  team  spent  more  time  on  COA  development  (on 
average).  Given  that  time  was  not  specified  as  an  MOE  for  the  planning  processes,  little  more 
should  be  inferred  into  the  above  observations  other  than  the  Integrated  process  appears  to  place 
a  greater  emphasis  on  MA  development  as  compared  to  the  Traditional  process. 
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3.3.4  ROLE  OF  LEAD  NATION 

During  each  vignette  a  “lead”  country  was  designated  for  both  the  Traditional  and 
the  Integrated  teams.  The  processes  followed  by  the  teams  during  the  LOE  appeared  to  embody 
different  perspectives  regarding  the  role  of  the  lead  country.  In  the  Traditional  process  “lead 
country”  was  synonymous  with  “leader”  and  throughout  the  LOE  team  members  examined  and 
reflected  on  the  topic  of  leadership/2  As  previously  noted,  lead  countries  often  had  the  primary 
role  during  MA  and  CO  A  development.  In  one  instance  a  Traditional  team  member  noted  that 
the  “. .  .(Lead  Nation)  went  into  a  nation  huddle  for  COA  development,  but  that  tends  to  be  the 
way  of  the  traditional  planning  process. . ..”  Some  participants  indicated  that  the  lead  country 
should  not  “. .  .give  out  too  much  of  its  own  responsibility  to  coalition  partners.” 

By  comparison,  in  the  Integrated  process  the  lead  country  operated  more  as  a 
manager  than  a  traditional  Lead  Nation.  When  the  US  operated  as  the  lead,  they  employed  the 
functional  cell  approach  in  order  to  integrate  the  activities  of  the  multinational  partners  during 
development  activities.  There  also  was  an  implicit  assumption  that  any  planning  activity 
conducted  by  the  lead  country  should  be  visible  to  the  entire  team. 

As  interesting  as  these  observations  are,  however,  one  must  be  cautious  about 
inferring  too  much  about  the  Integrated  process  and  the  underlying  RDO  concept.  Several  very 
important  factors  need  to  be  considered: 

•  COA  development  represents  a  window  of  a  larger  planning  cycle.  The  definition  of  that 
larger  planning  process  under  the  RDO  concept  is  still  under  development. 

•  The  structure  of  the  LOE  did  not  include  an  intermediate  command  level  between  the 
planning  cells  and  SACAPAC.  Based  on  conversations  with  members  of  the  US 
Integrated  team,  final  decisions  regarding  the  plan  would  be  made  at  the  next  higher 
level.  The  role  of  the  planning  cell  was  to  provide  recommendations  to  that  commander 
and  not  necessarily  make  “command  decisions.” 

•  Several  participants  noted  that  the  LOE  design  did  not  include  political  advisors  who 
would  provide  national  guidance  and  inject  national  priorities.  Lacking  this  potentially 
limiting  factor,  members  of  the  planning  cells  operated  almost  as  independent  agents. 
This  was  considered  to  be  an  artifact  of  the  experiment  and  operationally  unrealistic. 

Key  Findings 


The  processes  followed  by  the  teams  during  the  LOE  appeared  to  embody 
different  perspectives  regarding  the  role  of  the  lead  country.  In  the  Traditional  process  “lead 
country”  was  synonymous  with  “leader”  and  throughout  the  LOE  team  members  examined  and 
reflected  on  the  topic  of  leadership.  By  comparison,  in  the  Integrated  process  the  lead  country 
operated  more  as  a  manager  than  a  traditional  Lead  Nation.  There  also  was  an  implicit 
assumption  that  any  planning  activity  conducted  by  the  lead  country  should  be  visible  to  the 
entire  team. 


22  Based  on  feedback  compiled  from  Section  II  of  Vignette  Evaluation  Questionnaire. 
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3.3.5  IMPACT  OF  LEAD  NATION  ON  PROCESS 

Upon  the  request  of  the  multinational  participants,  the  US  cell  served  as  the  lead 
for  the  Integrated  team  for  the  entire  first  week  of  the  LOE  and  once  again  during  the  final 
vignette.  In  addition,  the  US  cell  developed  the  training  materials  and  business  rules  that 
provided  common  guidance  for  the  multinational  team  members.  By  contrast,  training  materials 
were  not  developed  for  the  Traditional  process  since  it  was  assumed  that  all  of  the  coalition 
countries  had  an  understanding  of  how  to  operate  under  the  Lead  Nation  concept.  The  lead 
assignment  for  the  Traditional  process  was  rotated  through  the  four  countries  so  that,  barring 
technical  problems,  each  country  would  serve  twice  as  lead. 

These  differences  in  training  and  lead  assignments  resulted  in  the  Integrated  team 
following  a  process  that  was  notably  more  stable  than  that  of  their  Traditional  counterparts.  The 
impact  of  doctrinal  and  cultural  preferences  were  more  pronounced  in  the  Traditional  team  as 
evidenced  by  the  fact  that  doctrinal  differences  was  often  cited  in  the  feedback  provided  by  the 
Traditional  team.23  A  participant  of  the  Analysis  workshop  offered  this  observation:  “National 
staff  training  and  processes  vary;  these  may  be  reflected  more  in  the  Traditional  analysis.  The 
consequence  of  developing  an  integrated  MN  staff  process  would  be  the  need  to  create  a 
common  training  process  across  all  nations.  This  would  have  national  and  inter-service  cultural 
implications.”  It  is  the  case  that  the  Traditional  team  refined  its  process  over  the  course  of  the 
LOE  and  developed  a  Standard  Operating  Procedure  that  incorporated  a  combination  of  the 
practices  introduced  by  the  different  lead  countries. 

Key  Findings 


Differences  in  training  and  lead  assignments  resulted  in  the  Integrated  team 
following  a  process  that  was  notably  more  stable  than  that  of  their  Traditional  counterparts.  The 
impact  of  doctrinal  and  cultural  preferences  was  more  pronounced  in  the  Traditional  team; 
doctrinal  differences  were  often  cited  in  the  feedback  provided  by  the  Traditional  team. 


3.3.6  EFFECTIVENESS  OF  INTER-COUNTRY  INTERACTION 

As  discussed  earlier,  the  two  planning  teams  exhibited  differences  in  pre-LOE 
training,  organizational  structure,  expectations  regarding  the  lead  country,  and  their  use  of 
Groove  tools.  Nevertheless,  for  each  vignette  both  teams  successfully  generated  a  COA, 
sometimes  in  the  face  of  significant  network  problems.  It  is  the  case  that  the  Integrated  team, 
more  than  the  Traditional  team,  indicated  that  technical  problems  at  times  degraded  the 
effectiveness  of  their  process.24  Team  members  noted  that  “degradation  of  (communications) 
reduced  (the)  ability  to  plan  interactively”  and  “without  it  ( the  collaboration  tool)  the 
collaboration  breaks  down  and  momentum  is  lost.”  The  planning  then  becomes  “highly 
traditional  in  nature.” 


23  Based  on  comments  compiled  from  responses  to  Section  II  of  Vignette  Evaluation  Questionnaire. 

24  Based  on  comments  compiled  from  Section  II  of  Vignette  Evaluation  Questionnaire. 
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Comments  from  the  Integrated  team  regarding  the  impact  of  network  problems 
reflect  the  underlying  assumption  that  maintaining  situational  awareness  of  team  activities  is 
critical  to  the  Integrated  process.  “Integrated  planning  relies  on  saturation  of  all  communications 
media  for  parallel  and  concurrent  efforts.”25  By  contrast,  some  members  of  the  Traditional  team 
indicated  that  at  times  there  was  too  much  data  exchange  among  team  members.  “In  the 
Traditional  process  a  lot  of  work  goes  on  in  the  background  as  the  lead  nation  develops  the 
COA.”27 


Another  dimension  of  inter-country  interaction  is  the  effectiveness  of  teams 
working  and  planning  in  a  distributed  environment.  As  reflected  in  the  summary  results  depicted 
in  Figure  3-6,  both  teams  indicated  that  the  distributed  environment  hampered  their  planning 
process  to  some  degree.  Yet  when  asked  whether  the  distributed  environment  made  it  more 
difficult  to  develop  and  maintain  situational  awareness  or  anticipate  the  needs  of  fellow 
members'  ,  the  responses  of  both  teams  reflected  ambivalence  and  significant  disagreement. 

Few  participants  indicated  that  it  was  more  difficult  to  “reach  consensus  over  decisions  in  a 
distributed  team”  (see  Figure  3-7). 


25  Comment  provided  in  Section  II  Vignette  Evaluation  Questionnaire. 

26  Sample  comments  include: 

“Continuous  data  feed  provided  by  GROOVE  from  unrecognizable  sources  seriously  hampered  the  intended  data 
flow.” 

“The  immense  amount  of  information  given  by  so  many  experts  can  hamper  own  consideration  flow  once  in  a 
while.  Too  many  opinions  are  not  always  helpful.” 

27  Source:  Team  Working  and  Planning  in  Coalition  Distributed  Teams  Survey,  response  to  Question  3.1. 

These  results  are  based  on  responses  collected  for  the  Team  Working  and  Planning  in  Coalition  Distributed  Teams 
Survey  administered  at  the  end  MN  LOE  I.  The  UK  analysis  team  developed  the  Coalition  Distributed  Teams 
Survey  as  a  part  of  their  human  factors  studies  conducted  during  MN  LOE  I.  The  UK  has  the  lead  for  the  analysis 
of  this  survey.  The  JHU/APL  analysis  team  compiled  summary  results  in  order  to  correlate  findings  from  this 
survey  with  observations  collected  from  other  data  sources.  Summary  charts  for  the  13  questions  contained  in  the 
survey  are  provided  in  Appendix  D. 

29  Summary  results  for  Questions  3.5  and  3.6,  the  Team  Working  and  Planning  in  Coalition  Distributed  Teams 
Survey,  are  provided  in  Appendix  D. 
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Question  3.1 


0%  20%  40%  60%  80%  100% 

%  of  Responses 


□  Disagree 

□  Neutral 

□  Agree 


Q  3.1  :  Monitoring  each 
other’s  actions  as  a 
distributed  team  was  more 
difficult  than  monitoring  each 
other’s  actions  in  a  co-located 
team. 


Q  3.3  :  It  was  more  difficult 
for  distributed  team 
members  to  identify  when 
other  team  members 
needed  assistance,  in 
comparison  to  co-located 
team  members. 


Figure  3-6:  Team  Working  and  Planning  in  Coalition  Distributed 
Teams  Survey:  Questions  3.1  and  3.3 


Question  3.8  :  It  was  more  difficult  to  reach  consensus  over  decisions  in  a 
distributed  team. 


Question  3.8 


%of  Responses 


□Disagree 
□  Neutral 
□Agree 


Figure  3-7:  Team  Working  and  Planning  in  Coalition 
Distributed  Teams  Survey:  Question  3.8 
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Key  Fin  dings 


During  each  vignette  both  teams  successfully  generated  a  COA,  sometimes  in  the 
face  of  significant  network  problems.  The  Integrated  team,  more  than  the  Traditional  team, 
indicated  that  technical  problems  at  times  degraded  the  effectiveness  of  their  process,  primarily 
because  maintaining  situational  awareness  of  team  activities  is  critical  to  the  Integrated  process. 
By  contrast,  some  members  of  the  Traditional  team  indicated  that  at  times  there  was  too  much 
data  exchange  among  team  members. 

The  teams  did  not  provide  a  strong  indication  the  distributed  environment  made  it 
more  difficult  to  develop  and  maintain  situational  awareness  or  anticipate  the  needs  of  fellow 
members.  Few  participants  indicated  that  it  was  more  difficult  to  “reach  consensus  over 
decisions  in  a  distributed  team.” 

3.3.7  EFFECTIVENESS  OF  INTRA-COUNTRY  INTERACTION 

For  both  processes  members  of  the  national  cell  communicated  with  each  other 
primarily  using  face-to-face  communications  vice  the  collaboration  tool.  The  US  Integrated  cell 
attempted  to  capture  national  cell  information  exchanges  in  Groove  but  even  in  their  case  a 
significant  amount  of  interaction  was  conducted  directly  person-to-person. 

Key  Findings 


For  both  processes  members  of  the  national  cell  communicated  with  each  other 
primarily  using  face-to-face  communications  vice  the  collaboration  tool. 

3.3.8  BALANCE  OF  SKILLS/EXPERIENCE/BACKGROUND  OF  TEAM 

MEMBERS 

During  post-LOE  discussions  some  control  cell  observers  commented  that  there 
was  disparity  in  the  skills,  experience  and  background  of  the  two  planning  teams.  When 
surveyed  during  the  Analysis  Workshop,  however,  observers  either  found  it  difficult  to  make  a 
knowledgeable  evaluation  of  this  criterion  or  disagreed  as  to  whether  the  teams  differed  on  this 
point.  It  is  the  case  that  both  planning  teams  indicated  they  lacked  logistics  expertise.  Also,  the 
US  Integrated  team  was  unique  in  that  some  members  had  worked  together  for  almost  a  year 
while  most  of  the  other  cells  were  working  together  for  the  first  time  during  MN  LOE  I. 

Key  Findings 


Both  planning  teams  indicated  they  lacked  logistics  expertise.  Also,  the  US 
Integrated  team  was  unique  in  that  some  members  had  worked  together  for  almost  a  year  while 
most  of  the  other  cells  were  working  together  for  the  first  time  during  MN  LOE  I. 
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3.3.9  ANALYSIS  WORKSHOP  ASSESSMENT  OF  PROCESS  CHARACTER¬ 

ISTICS  AND  SUMMARY  COMMENTS 

In  preparation  for  the  MN  LOE  I  Analysis  Workshop,  the  JHU/APL  analysis  team 
carefully  reviewed  the  extensive  data  compiled  for  the  eight  characteristics  defining  the 
Integrated  and  Traditional  planning  processes.  Developing  a  definitive  response  to  COI  #1  was  a 
challenge  since  the  two  processes  exhibited  both  similarities  and  differences.  Responding  to 
COI  #1  proved  to  be  a  challenge  for  the  participants  of  the  workshop  as  well.  Nine  participants 
(including  two  members  of  the  JHU/APL  team)  rated  each  of  the  characteristics  as  an  indicator 
that  the  processes  were  similar  or  different.  Summary  results  are  provided  in  Appendix  E.  The 
nine  participants  demonstrated  consensus  on  three  of  the  eight  characteristics:  role  of  lead, 
impact  of  lead,  and  inter-country  interaction.  The  two  processes  were  assessed  as  being  different 
on  the  first  two  points  and  similar  on  the  third.  Workshop  participants  disagreed  on  the 
remaining  five  points. 

In  summary,  the  comparison  of  the  two  planning  processes  suggests  that  the 
following  topics  require  further  consideration. 

•  The  Traditional  team  took  advantage  of  the  collaboration  tool  and  the  Traditional  process 
played  in  MN  LOE  I  was  more  integrated  in  nature  than  the  conventional  ‘stove-piped’ 
approach  typically  associated  with  the  Lead  Nation  concept.  Further  investigation  is 
required  to  assess  whether  given  the  availability  of  a  robust  collaboration  environment 
and  a  commonly  defined  process,  the  Traditional  process  might  naturally  evolve  into 
something  similar  to  the  Integrated  process. 

•  The  Traditional  and  Integrated  processes  followed  in  MN  LOE  I  demonstrated  significant 
similarities  and  differences,  and  consequently  a  definitive  comparison  of  the  two 
processes  proved  to  be  challenging.  The  two  processes  did  exhibit  noteworthy 
differences  in  the  areas  of  leadership  and  situational  awareness.  Issues  related  to 
leadership  have  potential  implications  for  doctrine,  procedures  and  training.  The 
infrastructure  implications  related  to  maintaining  continuous  situational  awareness  of 
team  activity  merits  further  study.  System  and  network  performance  data  collected 
during  MN  LOE  I  should  provide  good  insight. 

•  By  design,  planning  during  MN  LOE  I  was  primarily  conducted  at  the  operational  level. 

It  would  be  of  interest  to  examine  the  adaptation  of  the  Integrated  process  to  the  tactical 
level  (i.e.,  more  detailed,  real-time  planning). 

3.4  DETAILED  DISCUSSION  OF  COI  #2 

“Assuming  both  processes  are  equipped  with  the  same  collaboration 
capability,  does  the  quality  of  the  COAs  (completeness,  accuracy  and 
suitability)  produced  by  the  Traditional  and  Integrated  planning  processes 
differ  as  a  function  of  length  of planning  window  and  type  of  operation? 

The  examination  of  COI  #1  supports  the  conclusion  that  the  two  planning 
processes  demonstrated  in  MN  LOE  I  had  similarities  as  well  as  some  significant  differences. 
COI  #2  explores  whether  the  noted  differences  resulted  in  one  of  the  processes  consistently 
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producing  a  better  COA.  The  strategy  developed  to  conduct  this  assessment  was  to  have  LOE 
participants  and  senior  subject  matter  experts,  referred  to  as  ‘graybeards’,  critique  both  CO  As  for 
each  of  the  seven  vignettes.  The  participants  and  graybeards  would  evaluate  the  COA  in  terms 
of  three  MOEs:  completeness,  accuracy  and  suitability. 

After  the  first  vignette  LOE  participants  indicated  that  they  were  not  able  to 
properly  critique  the  COA  produced  by  the  other  team  due  to  time  limitations.  They  did  provide 
valuable  feedback  about  the  COAs  produced  by  their  own  team.  This  development  meant  that 
the  analysis  team  was  not  able  to  use  the  participants’  COA  evaluations  to  conduct  a  comparative 
assessment  of  the  two  processes.  Consequently,  the  examination  of  COI  #2  would  rely  on  the 
feedback  provided  by  the  graybeard  panel. 

Within  a  month  after  the  LOE,  the  graybeard  panel  was  provided  a  package  of 
materials  that  included  scenario  and  vignette  information,  Mission  Analysis  products,  Requests 
for  Information  and  Courses  of  Actions  for  each  of  the  seven  vignettes.  The  panel  was 
comprised  of  five  senior  experts  from  Australia  (2),  Germany  (1)  and  United  States  (2).  The 
planning  products  were  labeled  as  “Team  A”  and  “Team  B,”  Traditional  and  Integrated, 
respectively.  The  graybeards  were  then  asked  to  conduct  a  “blind  review”  using  a  common 
evaluation  questionnaire  (provided  in  Appendix  F).  The  panel  was  not  provided  indicators  as  to 
the  process  or  lead  country  that  generated  a  given  COA. 

Summary  results  of  the  graybeard  evaluation  are  provided  in  Figures  3-8  and  3-9. 
Given  the  fact  that  the  graybeard  panel  consisted  of  only  five  experts,  one  should  be  cautious 
when  assigning  statistical  significance  to  most  of  the  individual  vignette  scores.  Two  exceptions 
are  Vignettes  E  and  H,  where  a  significant  majority  of  the  experts  selected  the  COAs  produced 
by  the  Integrated  team.  The  strategy  for  evaluating  the  COAs  included  the  compilation  of  the 
total  number  of  deficiencies  identified  for  the  three  MOEs  (completeness,  accuracy  and 
suitability),  with  the  assumption  that  the  COA  with  the  least  number  of  deficiencies  would  be 
considered  better.  Though  the  graybeard  reviewers  provided  comments  explaining  their 
selections,  they  did  not  identify  deficiencies  at  a  level  that  would  allow  the  analysis  team  to 
generate  a  tally  for  comparative  purposes.  They  did  indicate  that  completeness  was  a  critical 
consideration  in  their  selections. 


Figure  3-8:  Summary  of  Senior  Subject  Matter  Expert  (Graybeard)  COA  Evaluations 
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Figure  3-9:  Summary  of  Graybeard  Selections 


Both  planning  teams  developed  a  COA  for  Vignettes  C  through  J.  Each  of  the 
reviewers  ( 1  through  5)  was  asked  to  indicate  if  they  preferred  one  of  the  two  COAs 
(Traditional  =  “A,”  Integrated  =  “B”),  or  otherwise  rate  them  as  equal  (“Eq”). 

In  the  case  of  Vignettes  E,  J,  and  I,  a  reviewer  rated  the  two  COAs  as  comparable, 
i.e.,  ‘Eq.”  Both  teams  were  assigned  a  half-vote  for  these  vignettes. 

The  following  points  summarize  the  graybeard  feedback: 

•  In  several  instances  the  experts  noted  little  difference/preference  between  the  two  COAs 
though  when  asked  to  indicate  a  preference  they  selected  “Team  B.” 

•  They  suggested  that  the  commander  would  send  even  the  preferred  COAs  back  to  the 
planners  for  more  work  on  specifics. 

•  In  some  cases  the  experts  recommended  that  elements  of  the  COA  not  selected  should  be 
incorporated  in  next  version  of  the  selected  COA. 

•  The  experts  noted  equivalent  deficiencies/problems  in  both  COAs. 

•  The  final  selections  were  based  primarily  on  completeness.  Suitability  was  rarely 
discussed  and  accuracy  was  apparently  considered  adequate  since  few  specific  errors 
were  noted. 

The  participants  of  the  Analysis  Workshop  reviewed  the  graybeard  results  to 
identify  the  reason(s)  why  the  graybeard  panel  expressed  an  overall  preference  for  the  COAs 
produced  by  the  Integrated  team.  Type  of  operation  and  length  of  planning  window  were 
considered;  a  summary  of  these  characteristics  for  the  seven  vignettes  is  provided  in  Table  3-3. 
As  previously  stated,  all  the  vignettes  were  based  on  MOOTW  vice  warfighting  situations.  One 
participant  of  the  analysis  workshop  conjectured  that  “the  vignettes  were  reasonably  similar  and 
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therefore  the  integrated  team  was  better  at  coming  up  with  a  COA  given  the  restricted  (low 

on 

intensity)  operations.”  Other  than  this  general  observation,  an  examination  of  the  seven 
vignettes  does  not  suggest  an  overall  trend  that  would  readily  explain  the  graybeard  selections. 
Both  Vignettes  E  and  H  were  six  hours  in  length  and  addressed  piracy  situations.  However,  the 
Integrated  process  was  rated  better  for  both  short  (6  hours)  and  longer  (9  hours)  vignettes.  Also, 
for  Vignette  J,  another  piracy  operation,  the  CO  As  yielded  similar  ratings  for  the  two  teams. 


Table  3-3 

Summary  of  MN  LOE  I  Vignettes 


Vignette 

Duration  of 
Vignette  (hrs) 

Operation 

Description 

C 

6 

Humanitarian  Assistance 

Typhoon;  Vietnam 

E 

6 

Protection  of  Shipping; 

Counter  piracy 

Merchant  ship  flagged  to  one  of  the 
coalition  members;  Malacca  Straits 

F 

10 

Recovery  Operations; 

Counter  Piracy 

Hostage  situation  on  cruise  ship; 
Semarang,  Indonesia 

G 

6 

Noncombatant  Evacuation 
Operations 

Citizens  of  coalition  countries;  Ache 
Province,  Indonesia 

H 

6 

Protection  of  Shipping; 
Counterrorism,  Counter  piracy 

Japanese  ship  carrying  radioactive 
materials,  terrorist  threat 

I 

8 

Humanitarian  Assistance 

Cyclone;  Papua  New  Guinea 

J 

9 

Recovery  Operations; 

Counter  piracy 

MV  WEN  ZHO  seized;  Bikini  Atoll, 
Marshall  Islands 

Another  possible  explanation  unrelated  to  the  actual  vignettes  is  that  the 
Integrated  team  was  able  to  better  retain  corporate  knowledge  and  leverage  off  previous  planning 
efforts:  “  . . .( 1)  the  US  led  most  of  the  time,  so  the  other  countries  learned  the  style  early  on  and 
could  then  spend  proportionally  more  time  working  on  MA/COA  development  and  (2)  the 
Traditional  team  had  more  dead  time  within  the  experimental  timeline,  so  the  Integrated  Team 
had  a  higher  level  of  manpower  actively  working  the  problem. . . .  Could  be  a  combination  of 
both  of  these,  and  other  factors  was  well.”  Yet  another  suggestion:  “. .  .did  the  Integrated  team 
receive  more  positive  votes  due  to  the  COA  development  as  a  team  versus  the  Traditional 

T9 

approach  which  restricted  input,  or  rather  filtered  it  through  the  lead  nation?” 


30  Appendix  E:  MN  LOE  I  Analysis  Workshop 

31  ibid 

32  :u;a 
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Key  Fin  dings 


Overall,  the  graybeards  marginally  preferred  the  COAs  produced  by  the 
Integrated  team,  citing  completeness  as  a  critical  factor.  Given  the  small  size  of  the  graybeard 
panel  (five),  statistical  significance  cannot  be  assigned  to  most  of  the  individual  vignette  scores. 
Two  exceptions  are  Vignettes  E  and  H,  where  a  significant  majority  (4.5  and  5,  respectively)  of 
the  experts  selected  the  COAs  produced  by  the  Integrated  team. 

The  analysis  of  the  graybeard  evaluations  did  not  produce  a  definitive  explanation 
of  the  factors(s)  causing  the  graybeards  to  prefer  the  Integrated  COAs.  The  examination  of  the 
types  of  operations  and  lengths  of  planning  window  represented  in  the  seven  vignettes  did  not 
suggest  an  overall  trend  that  would  readily  explain  the  graybeard  selections.  A  workshop 
participant  suggested  that  the  “Integrated  team  was  better  at  coming  up  with  a  COA  given  the 
restricted  (low  intensity)  operations.”  This  is  an  interesting  observation  that  requires  further 
study. 


Another  factor  that  merits  further  examination  is  the  fact  that  the  Integrated  team 
followed  a  process  for  which  they  recently  received  training  and  that  was  played  consistently 
throughout  the  experiment.  Overall,  this  may  have  allowed  the  Integrated  team  to  better  retain 
corporate  knowledge  and  leverage  off  previous  planning  efforts. 


3.5  DETAILED  DISCUSSION  OF  COI  #3 


“Can  collaborative  planning  be  conducted  successfully  in  a  distributed 
environment?  ” 

Following  the  in-depth  analysis  conducted  to  address  COI  #1  and  COI#2,  it  is 
clear  that  the  MN  LOE  I  did  demonstrate  that  collaborative  planning  can  be  conducted  in  a 
distributed  environment.  First  of  all,  both  teams  were  able  to  successfully  develop  a  COA  for 
each  vignette,  sometimes  in  spite  of  significant  technical  problems.  The  comments  provided  in 
the  Vignette  Evaluation  Questionnaires  support  the  statement  that  the  LOE  participants  were  able 
to  persevere  and  successfully  conduct  planning.  Some  of  the  participants  indicated  that  by  the 
end  of  the  LOE  the  planning  processes,  especially  the  Integrated  process,  were  working  well.  As 
previously  cited,  the  participants  indicated  in  their  responses  to  the  distributed  teams  survey  that 
they  were  able  to  maintain  situational  awareness  and  work  successfully  with  team  members  who 
were  not  co-located.  Also,  the  participants  responded  that  a  distributed  environment  did  not  pose 
a  significant  obstacle  to  achieving  consensus. 

When  polled  during  the  Analysis  Workshop  as  to  whether  MN  LOE  I 
demonstrated  that  collaborative  planning  can  be  conducted  successfully  in  a  distributed 

33 

environment,  the  attendees  overwhelmingly  responded  in  the  affirmative. 


33  Data  from  the  Analysis  Workshop  are  provided  in  Appendix  E. 
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Even  with  these  positive  results,  the  potential  adverse  impact  of  computer  system 
problems  and  network  congestion  on  the  planning  processes  should  not  be  ignored  nor 
minimized.  Additional  study  is  required  to  specify  the  system  requirements  associated  with 
supporting  distributed  planning,  especially  at  the  level  represented  in  the  Integrated  process.  The 
network  problems  experienced  during  MN  LOE  I  adversely  impacted  both  planning  processes, 
though  it  is  very  difficult  to  assess  to  what  degree,  except  in  the  case  of  Germany  who  for  the 
most  part  was  excluded  from  participation  due  to  poor  network  performance.  Of  special  interest 
are  the  requirements  associated  with  maintaining  continuous  situational  awareness  of  planning 
team  activity. 

Another  critical  component  of  planning  is  the  exchange  of  classified  data.  By 
design  this  was  not  addressed  in  MN  LOE  I  but  will  need  to  be  explored  in  future  LOEs. 
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Section  4 

RECOMMENDATIONS 

4.1  INTRODUCTION 


The  feedback  collected  from  participants  and  observers  of  MN  LOE  I  and  the 
attendees  of  the  Analysis  Workshop  provided  the  analysis  team  with  many  valuable 
recommendations  for  improving  future  LOEs. 


4.2  LOE  PARTICIPANT  RECOMMENDATIONS 


The  participants  of  MN  LOE  I  had  many  opportunities  to  provide  feedback  and 
offer  recommendations  for  improving  the  LOE  process.  The  participants  completed  Vignette 
Evaluation  Questionnaires  after  each  vignette  and  the  Team  Working  and  Planning  in  Coalition 
Distributed  Teams  Survey  at  the  end  of  the  LOE.  The  comments  of  the  participants  pertaining  to 
experiment  structure  and  design  and  the  Groove  application  are  summarized  below.  Relevant 
comments  added  by  the  JHU/APL  analysis  team  are  highlighted  in  italics. 

4.2.1  EXPERIMENT  DESIGN  AND  STRUCTURE 

LOE  participants  offered  the  following  recommendations  pertaining  to 
experiment  design  and  structure: 

•  A  robust  network  and  better  stress  testing  of  the  system  prior  to  the  experiment  are 

critical  to  successful  LOEs. 

•  Vignettes  must  be  sufficiently  challenging. 

■  Participants  expected  updates  to  the  operational  situation  would  be  provided  at  the 
beginning  of  each  vignette.  Also,  permutations  to  the  vignettes  during  play  added  a 
lot  of  realism. 

■  Participants  indicated  that  pre-LOE  scenario  training  materials  should  be  better 
aligned  with  the  situations  played  during  the  LOE.  The  need  for  unclassified 
vignettes  necessitated  the  selection  of  sites  outside  the  core  Pacifica  military 
operation  while  the  scenario  materials  provided  in  the  training  package  focused  on  the 
Pacifica  situation. 

■  Participants  indicated  a  strong  preference  for  warfighting  situations.  An  issue  that 
requires  immediate  investigation  is  whether  this  preference  can  be  accommodated  in 
an  unclassified  environment  or  whether  future  LOEs  will  need  to  be  conducted  in  a 
classified  environment.  This  point  is  addressed  again  in  Sections  4.3  and  4.4. 

•  Intelligence  Preparation  of  Battlespace  must  be  adequate  to  support  vignette  play. 

■  This  point  is  related  to  broader  issues  pertaining  to  Operational  Net  Assessment 
(ON A).  The  players  expected  that  a  well-developed  ONA  would  be  available  at  the 
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start  of  play.34  Participants  also  noted  that  the  research  tools  (BrainEKP  and 
Autonomy)  were  not  adequately  initialized  to  serve  as  viable  ONA.  Development  of 
a  realistic  ONA  for  use  in  follow-on  LOEs  requires  the  commitment  of  significant 
resources  during  LOE  development. 

•  More  training  is  required  to  adequately  prepare  the  participants  for  using  the  tools, 
surveys  and  processes. 

•  LOE  Command  and  Control  structure  should  incorporate  critical  real  world  elements 
such  as  political  advisors. 

•  A  higher  level  of  activity  on  the  part  of  the  white  cell  is  necessary. 

■  Participants  cited  the  need  for  greater  consistency  in  the  guidance  provided  by  the 
white  cell  as  well  a  higher  level  of  interaction  during  play.  Meeting  these  requests 
would  likely  require  increasing  the  number  of  personnel  assigned  to  the  LOE 
Commander’s  (e.g.,  SACAPAC  in  MN  LOE  I)  staff 

4.2.2  GROOVE  ENHANCEMENTS 

The  LOE  participants  offered  specific  recommendations  regarding  enhancements 

to  Groove: 

•  Improve  the  word-processing  capabilities  of  the  Groove  tools  (e.g.  include  spell  check 
capability). 

•  Improve  the  capabilities  and  perfonnance  of  the  whiteboard. 

•  Improve  Notepad  to  better  support  interactive,  collaborative  development  activities. 
Control  of  the  scroll  feature  was  especially  noted  as  a  desirable  feature. 

•  Improve  messaging  by  adding  email-like  features.  In  particular  participants  wanted  to 
know  everyone  who  was  included  on  the  distribution  of  a  given  message. 

•  Incorporate  a  print  capability  into  the  Groove  tools. 

•  Provide  better  means  of  archiving  data  from  Groove  and  initializing  shared  spaces. 

4.3  ANALYSIS  WORKSHOP  RECOMMENDATIONS 


The  MN  LOE  I  Analysis  Workshop  was  held  at  JHU/APL  with  attendees  from  J9, 
UK,  Canada  and  Germany.  The  attendees  of  the  workshop  concurred  with  the  recommendations 
of  the  LOE  participants  and  offered  the  following  additional  recommendations: 

•  Many  attendees  felt  strongly  that  future  MN  LOEs  should  include  political  advisers. 


34 

The  following  represents  a  typical  comment  on  this  matter:  “The  information  provided  in  the  Warning  Order  is 
only  sufficient  if  the  vignette  is  supported  with  solid,  prepared  information.  If  this  is  not  provided,  the  planning 
team  must  do  one  before  the  MA.” 

35  The  following  represents  a  typical  comment  on  this  matter:  “In  reality,  the  domestic  political  considerations  have 
far  greater  influence  and  these  factors  are  what  makes  multi-national  planning  very  difficult.  The  only  way  to 
simulate  this  is  for  a  local  exercise  controller  to  require  a  participant  to  act  in  a  certain  way  during  the  shaping  of  a 
COA.” 
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■  This  recommendation  precipitated  comments  from  other  attendees  that  the  inclusion 
of  political  advisors  and  other  real  world  elements  may  greatly  expand  the  scope  of 
the  limited  objective  experiments. 

•  Several  attendees  indicated  that  LOEs  need  vignettes  that  are  “complex,  warfighting  and 
classified”. 

•  A  specific  point  made  regarding  the  ONA  was  that  LOE  participants  should  be  provided 
a  common  set  of  maps  compiled  specifically  for  each  vignette. 

•  It  was  suggested  that  a  separate  Groove  space  be  established  for  sharing  lessons  learned 
regarding  Groove  operations  during  vignette  play. 

•  LOE  surveys  should  be  approved  prior  to  the  start  of  the  LOE  and  data  collection  efforts 
should  be  integrated  into  a  single  survey.  A  web-based  user  interface  would  help  to 
ensure  the  integrity  of  the  data  collected. 

•  Disparity  in  skill  and  experience  levels  across  the  international  teams  should  be 
eliminated.  The  method  to  do  this  needs  to  be  addressed  during  LOE  planning,  when  the 
LOE  staffing  requirements  are  established. 

•  Capability  requirements  for  collaboration  tools  should  be  investigated  more  in  the 
upcoming  LOEs. 

•  Each  LOE  needs  an  Information  Manager  who  will  monitor  network  and  system 
performance  and  oversee  system  initialization  prior  to  vignette  play  and  data  archival 
activities. 

•  The  LOE  white  cell  should  include  a  Flag  level  commander  who  would  provide  guidance 
and  feedback. 

■  The  presence  of  this  commander  would  serve  to  ‘raise  the  bar’  on  the  performance 
expected  from  participants. 

•  There  was  considerable  debate  as  to  whether  the  two-process  structure  (Traditional  and 
Integrated)  should  be  maintained  in  follow-on  MN  LOEs. 

■  The  point  was  made  that  in  order  for  JFCOM  to  get  buy-in  from  multinational 
partners  for  migrating  to  new  operational  concepts,  it  is  critical  to  establish  an 
irrefutable  case  and  this  can  be  done  by  continuing  to  compare  traditional  concepts 
against  new  concepts  in  the  MN  LOEs. 

•  Establish  an  LOE  Analysis  Working  Group  that  would  “incorporate  analysis  from  (all) 
countries”  and  coordinate  activities  during  the  planning  and  analysis  phases  of  the  LOE. 

•  The  role  of  observers  during  LOEs  is  critical;  the  control  cell  should  be  appropriately 
staffed  with  observers. 
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4.4  JHU/APL  RECOMMENDATIONS 


Overall,  the  JHU/APL  team  endorses  the  recommendations  provided  by  the  LOE 
participants  and  attendees  of  the  Analysis  Workshop.  Further  discussion  on  some  of  these  points 
and  additional  recommendations  are  provided  below: 

Areas  Requiring  Further  Investigation 

•  Several  topics  worthy  of  additional  investigation  were  identified  during  the  analysis  of 
MN  LOE  I.  It  is  recommended  that  J9  consider  exploring  the  following: 

■  Given  the  availability  of  a  robust  collaboration  environment  and  a  commonly  defined 
process,  might  the  Traditional  process  naturally  evolve  into  something  similar  to  the 
Integrated  process  and  if  not,  what  are  the  major  obstacles?  In  particular,  the  role  of 
the  lead  nation  as  a  leader  needs  to  be  further  explored  in  the  Integrated  process. 

■  What  are  the  infrastructure  implications  associated  with  maintaining  continuous 
situational  awareness  of  team  activity  as  demonstrated  in  the  Integrated  process? 

Planning  for  Future  MN  LOEs 

•  A  planning  roadmap  for  Olympic  Challenge’ 04  would  aid  in  ensuring  that  the  objectives 
of  the  individual  LOEs  support  the  larger  objectives  defined  for  Olympic  Challenge  ’04. 

•  It  also  of  paramount  importance  to  start  planning  for  the  LOEs  as  soon  as  possible,  even 
if  the  level  of  effort  during  the  initial  stages  is  minimal.  An  early  start  is  necessary  to 
ensure  that  critical  issues  and  concerns  are  addressed  early  in  the  process,  thus  increasing 
the  potential  for  successful  resolution. 

LOE  Structure 

•  J9  should  retain  the  distributed  environment  in  future  LOEs. 

•  J9  should  take  under  serious  consideration  the  recommendation  to  retain  the  2-process 
structure  (new  vs.  conventional)  in  future  LOEs.  The  MN  LOEs  are  critical  to  examining 
and  demonstrating  the  value  and  viability  of  proposed  RDO  concepts.  If  a  comparison  of 
new  concepts  against  more  ‘traditional’  concepts  is  not  part  of  the  LOE  structure,  then  a 
comparative  assessment  will  need  to  be  obtained  through  some  other  means. 

•  The  JHU/APL  analysis  team  agrees  that  keeping  participants  actively  engaged  at  a 
challenging  operational  tempo  is  important  to  successful  LOEs. 

•  The  MN  LOE  I  experience  demonstrated  that  more  complex  vignettes,  with  continuous 
updates  to  the  operational  situation,  should  be  incorporated  into  future  LOEs.  We 
strongly  endorse  the  recommendation  that  a  Flag  Level  commander  be  included  in  the 
white  cell  in  the  interest  of  ‘raising  the  bar’  for  the  participants. 

•  Access  to  classified  data  and  challenging  vignettes  are  two  related  yet  separate  issues. 

MN  LOE  I  participants  indicated  the  desire  for  more  challenging  vignettes  and,  as  stated 
above,  this  request  needs  to  be  addressed.  Whether  this  translates  into  war  fighting, 
classified  vignettes  needs  to  be  examined  and  not  taken  as  an  ‘a  priori’  assumption.  On  a 
separate  point,  multinational  partners  have  indicated  a  desire  to  move  the  LOEs  into  the 
classified  environment,  believing  that  this  is  more  representative  of  the  operational 
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environment  in  which  the  RDO  concepts  will  be  practiced.  This  request  also  needs  to  be 
addressed. 

•  LOE  procedures  should  be  formalized  and  publicized  to  all  participants.  This  is  related 
to  the  need  for  more  extensive  and  common  training.  A  workshop  participant  suggested 
that  a  training  video  would  be  helpful. 

•  The  concepts  being  examined  in  the  LOE  series  need  to  be  established  early  in  the 
planning  process.  Otherwise,  the  experiment  design  may  not  be  aligned  with  the  Critical 
Operational  Issues  that  need  to  be  examined. 

Role  of  Multinational  Partners 


•  J9  should  encourage  the  multinational  partners  to  play  a  more  active  role  in  scenario 
development  and  experiment  design. 

Staffing  Requirements 

•  A  designated  LOE  Information  Manager  and  a  sufficient  staff  of  observers  are  critical  to 
the  smooth  conduct  of  LOEs.  The  overall  analysis  strategy  should  incorporate  observer 
activities. 

•  A  design/analysis  working  group  should  be  established  early  in  the  planning  phase  and 
the  LOE  management  team  should  form  a  steering  group  that  includes  representation 
from  the  scenario  and  design/analysis  working  groups.  This  structure  will  facilitate  the 
coordination  of  parallel  yet  potentially  separate  planning  efforts. 
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Excerpt  from  J9  “ SJC2E  Briefings  for  MN  LOE I  Trainers  ” 


Collaboration: 

The  need  to  operate  inside  an  adversary’s  decision  cycle  will  require  that  the  SJC2E  assist  in 
executing  a  more  simultaneous,  collaborative  planning  and  execution  process.  The  need  for 
unity  of  effort  to  ensure  the  application  of  decisive  effects  when  and  where  needed  requires 
superior  shared  battlespace  understanding,  awareness  of  the  current  situation,  and  a  common 
understanding  of  the  commander’s  intent.  One  of  the  main  functions  of  the  SJC2E  prior  to  the 
crisis,  and  the  MNF  during  the  crisis,  is  to  ensure  full  integration  of  MNF  with  I  AC  and  non¬ 
government  capabilities  across  all  domains.  Using  USJFCOM  J9’s  Joint  Interactive  Planning 
(JIP)  process,  the  MNF  and  component  planning  cells  execute  distributed,  collaborative,  parallel 
planning  with  higher  headquarters  and  other  key  governmental  agencies.  Collaboration  provides 
the  means  to  achieve  the  integration  required  of  RDO. 

Information  technology  will  provide  tools  for  COA  development  and  analysis  that  will  shorten 
planning  times  and  allow  dynamic,  continuous  plan  modification  during  execution.  These  same 
tools  will  support  realistic  mission  rehearsal  and  training.  Commanders  will  use  collaboration 
tools  to  confer  with  other  commanders,  their  distributed  staffs,  and  SME’s  for  planning  and 
battle  management. 

The  integrated  nature  of  future  operations  requires  a  MNF  headquarters  designed  around 
mission-tailored  collaborative  networks.  These  virtual,  collaborative  networks  include  not  only 
permanent  members  of  the  MNF  staff  but  also  representatives  from  supporting  agencies. 
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Mission  Analysis  Summary 


Joint  Concept  Development  and  Experimentation 


Determine 
Known  facts 


Develop 

Assumptions 


Analyze 
CINC  Mission 
and  Intent 


•  JIPB 

•  Status  of  forces 
•Time  available 
'Asset  availability 
•Lift 


•  Used  to  replace 
missing  or  unknown 
facts 

•  Essential  for 
planning  to  continue 

•  Characteristics 

-  Logical 

-  Realistic 


•  CINC  planning 
directives 
.  “Why”  from 
mission 


•Things  you 
must  do 
•Things  you 
must  not  do 
•  Others 


•Friendly 
•  Enemy 


Tasks 

•  Specified 

•  Implied 

•  Essential 


success 


■  Mission  What  might 

requirements  vs.  rjsk  mission 
available  assets 

■  Determine 
possible  shortfalls 
&  request 
augmentation 


Set  of 
conditions 
which  will 
achieve  the 
purpose  of 
the  mission 


•Mission  analysis  should  be  continuous.  As  the  situation  changes  so  may  certain  aspects  of  the 
mission.  Revisit  what  you’ve  done  when  major  events  have  occurred. 

•Review  your  Facts!  CINC  Assumptions  are  facts  for  MNFs  during  planning  unless  otherwise 
answered. 

•Assumptions  can  be  an  important  source  of  your  initial  RFIs. 

•Share  JIPB  with  components! 

•End  states:  Consider  strategic,  establish  operational,  give  guidance  on  tactical. 

•Products  from  this  phase  are:  Approved  MSN  Statement  and  CDRs  Planning  Guidance. 
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Commander’s  Guidance 


.  '‘tint  Concept  Development  and  Experimentation 


Guidance  to  Staff 
and  Subordinates 

Restated  Mission 
Warning  Order 


*  Initial  guidance 

-  Feedback  on  staff’s  mission  analysis 

-  Initial  Commander’s  Intent 

-  CCIR 

-  Other  guidance  to  focus  planning 

*  Subsequent  guidance 

-  Amplifies  /  modifies  initial  guidance 

-  Results  from  staff  engaging  commander 

*  Criteria  for  comparison 


•Send  out  at  the  earliest  time  to  help  subordinate  staffs  meet  timelines. 

•Purpose  should  be  to  focus  the  staff  process. 

•The  CDRs  Intent  can  provide  criteria  for  COA  development  and  comparison  later,  e.g.  flexibility, 

speed,  mass,  footprint  size,  force  closure  times,  etc. 

•Provide  the  CCIR  in  order  to  facilitate  information  management  and  collection  efforts.  The  decision 
making  process  is  streamlined.  It  will  come  from  JIPB  and  is  refined  later  during  wargaming! 

•PIR  (enemy  focus)  EEFI  (friendly  info  that  must  be  protected)  FFIR  (info  about  your  own  forces) 


A-3 


COA  Development 


lint  Concept  Development  and  Experimentation 


Staff 
Estimates 


Statements, 
Sketches,  &  Task 
Organization 
Warning  Order 


•  Review  tasks,  guidance,  intent,  and  mission 

•  Brainstorm 

•  Test  draft  COA 

•  Determine  C2 

•  Prepare  COA  statement  and  sketch 

•  Brief  CMNF 

•  Staff  estimates  are  ongoing 


COAs  must  pass  the  test  of; 

"SUITABLE  -  must  accomplish  mission  and  comply  w /  CINC  guidance. 

"FEASIBLE  -  must  accomplish  mission  within  the  established  time,  space  and  resource  constraints. 

"ACCEPT ABLE  -  must  balance  risk  with  advantage  gained  by  executing  a  particular  COA. 

"DISTINGUISHABLE  -  each  COA  must  be  significantly  different  from  the  others. 

"COMPLETE  -  incorporates  major  tasks  and  operations  to  be  accomplished  to  include;;  forces 
required,  logistics  concept,  employment  concept,  reserve  force  concept,  time  estimates  for  reaching 
termination  objectives,  and  end  state. 

"Must  develop  a  rough  TPFDD  to  look  at  feasibility. 

"Look  at  phasing  to  focus  your  efforts. 
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The  Plans  Group  is  to  provide  the  MNF  commander  with  a  more  integrated  approach  to  planning  focused  in 
future  operations.  It  also  forms  the  basis  for  an  expanded,  interactive  Joint  Planning  Center  (JPC)  linked 
through  the  collaboration  network  to  conduct  operational  planning. 

The  Operations  Group  focuses  on  current  operations.  They  perform  the  integrating  and  synchronizing 
function  for  current  operations  by  monitoring  on-going  actions  and  assisting  the  Information  Superiority  Group 
in  assessing  the  outcomes  and  effects  achieved  as  required  to  achieve  the  Commander’s  intent  and  guidance. 
When  necessary,  the  Operations  Group  will  revise  or  direct  changes  to  an  existing  order  based  on  the  situation, 
through  issuance  of  changes  to  that  order  .  A  close  relationship  with  the  Plans  and  Information  Superiority 
Group  is  required  to  ensure  that  future  operations  encompass  and  adequately  address  the  assessments  of  on¬ 
going  operations. 

The  Information/Knowledge  Management  Group  is  that  element  of  the  SJC2E  that  provides  knowledge 
management  and  technical  support  to  MNF  planning  and  execution. 

On  the  technical  side,  this  Group  serves  as  both  the  network  planning/control  section  and  the  technical  support 
section,  ensuring  that  the  SJC2E  has  the  connectivity  and  systems  availability  necessary  to  support  operations. 
The  knowledge  management  section  supports  the  SJC2E  by  facilitating  the  dynamic  creation,  discovery, 
manipulation  and  provision  of  the  right  information,  to  the  right  people,  at  the  right  time,  in  the  right  format. 
Properly  planned  and  executed,  knowledge  management  allows  decision  superiority  through  a  culture  of 
information  sharing.  This  Group  may  be  utilized  as  either  augmentees  or  as  a  “plug”  to  an  existing  operational 
headquarters. 

Reach-Back  and  Augmentation 

The  MNF  headquarters  integrates  support  from  distributed  staffs,  SME’s  and  COE  for  intelligence,  planning 
expertise,  effects,  administration,  and  logistic  support.  This  reach-back  capability  allows  the  MNF  headquarters 
to  operate  from  geographically  distributed  locations  to  best  support  the  mission,  situation  and  commander 
requirements.  The  MNF  is  not  only  augmented  by  the  special  capabilities  of  the  SJC2E,  but  must  be  also 
augmented  by  designated,  trained,  and  exercised  augmentation,  “plugs”,  and  liaisons  capable  of  rapid 
deployment  to  join  the  staff  or  linked  by  collaborative  planning  tools.  Functional  “plugs”  might  include:  rear 
area  protection,  Civil  Military  Operations  Center  (CMOC);  liaison  might  include  Joint  Warfare  Analysis  Center 
(JWAC)  normally  located  at  the  CINC,  TRANSCOM,  National  Intelligence  Support  Team  (NIST)  normally 
located  at  the  CINC,  and  component  command  liaisons;  augmentees  would  be  tailored  to  augment  the  existing 
capabilities  of  the  SJC2E  and  existing  MNF  headquarters  staff.  Depending  upon  the  situation,  it  may  also  be 
appropriate  to  include  liaison  elements  from  our  regional  allies  and  key  coalition  partners  to  aid  in  coalition 
building  and  maintenance. 
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SJC2E  MN  Plans  team 


Joint  Concept  Development  and  Experimentation 


Collaborative 

Environment 


Control  Cell  Responsibility: 

ScenarioA/ignette 

Development 

ScenarioA/ignette  Execution 
Works  as  CINC/  NCAs  or  other 
outside  agencies 
Reach  Back  for  I  A,  CINC 
Supervise  and  manages  Info 
exchanges 


The  Joint  Planning  Center  (JPC)  conducts  Crisis  Action  Planning  and  performs  future  operations  planning. 
The  JPC  participates  in  support  of  the  CINC  J5  in  CONPLAN  development  pre-crisis  and  during 
contingency  operations.  It  is  the  focal  point  for:  orders  development,  mission  refinement,  planning 
guidance,  COA  development/  analysis  and  OPORD  coordination,  makes  recommendations  to  Joint 
Coordination  Board(JCB).  A  virtual  collaborative  planning  network,  the  JPC  is  composed  of  planning 
representatives  from  the  MNF  headquarters,  CINC’s  staff,  Multi-National  Commands,  subordinate 
components,  and  other  agencies  functionally  oriented  to  the  MNF  commander’s  mission.  After  receiving 
specific  guidance  from  the  JCB,  the  JPC  employs  this  collaborative  network  on  a  continuous  basis  to 
develop  orders  as  required  to  meet  the  MNF  commander’s  desired  effects  and  ultimately  the  CINC’s 
objectives.  The  developed  order  is  then  forwarded  to  the  Joint  Operations  Center  for  review  and  release 
after  approval  by  the  MNF  commander  or  his  designated  representative.  The  Plans  Director  assumes 
leadership  of  the  JPC  within  the  MNF. 
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MN  LOE  Plans  Team 


Joint  Concept  Development  and  Experimentation 


DE 

UK 

(3) 

(3) 

1  Plans  /" 

■si  Plans 

1  Inter 

Lintel 

i  lo4 

/  usV 

1  Kog 

(5) 

AS 

1  Plans 

1  Intel 

1  Log 

Plans  director  distributes  billets 
\  based  on  expertise  and  experience 
to  fill  the  SJC2E  plans  team  roster 

US  planners; 

SOF/Civil  affairs 

Ground 

Air 

Naval 

Red  teamer/Force  Protection 


SJC2E  MN  Plans  team 


Joint  Concept  Development  and  Experimentation 


Contbil 


ant 


From  14  to  17 


Plans  Team  (17) 

•Plans  Director 

•Intel  Analysts  (2) 

•Planners  (5) 

•Logistics  Coordinator 

•Logistics  Deployment  Planner 

•Logistics  Sustainment  Planner 

•Blue/Red  Cell  (2) 

•Political  /  Military  Planner 

•Civil  Affairs  Planner 

•Ops  Law  Planner 

•Force  Protection  Planner 
(TBM/WME) 


A-7 


Integrated  Planning  Process 


Joint  Concept  Development  and  Experimentation 


CINC’s 

Mission 

Mission 

Analysis 

MNJPC 


Inside  the 

Collaborative 

Network 


The  MNJPC  (Plans  Team)  is  in  a  collaborative  network  that  includes  the  CINC,  Components  and  the 
IAC/COE. 

CINC,  IAC/COE  and  components  functions  and  responses  are  conducted  by  the  White  Cell.  The  Plans 
Team  is  the  portion  of  the  CJFHQ,  which  is  replicated  in  this  experiment.  The  other  parts  of  the 
CJFHQ:  OPS,  Information  Superiority,  Knowledge/Information  Management  and  Command  are  either 
not  applicable  to  this  LOE  or  are  replicated  by  the  White  Cell. 

Business  rules  are  written  to  provide  a  framework  upon  which  the  collaborative  sessions  will  operate. 
Team  leaders  from  the  nation  designated  as  the  Lead  Nation  or  the  CJFHQ,  whichever  is  the  case,  play 
a  large  part  in  conducting  smooth  and  coherent  collaboration  sessions.  The  Business  Rules  that  follow 
address  who  is  performing  which  functions,  why  they  perform  them,  and  what  tools  are  used  in  which 
way.  Business  rules  also  address  what  the  final  form  of  the  product  should  look  like.  There  are 
business  rules  that  also  address  how  the  different  functions  of  the  team  work,  both  internally  focused  as 
well  as  across  the  spectrum  of  the  entire  scenario.  Business  rules  have  owners/advocates  (e.g., 
individuals  in  charge  of  groups/cells.  Information  Managers,  etc.)  who  are  responsible  for  ensuring  the 
rules  are  understood,  followed,  and  modified  as  necessary,  with  the  changes  communicated  to  the 
participants.  The  business  rules  provide  a  common  starting  point.  Team  leaders  may  alter  the  rules  as 
required  by  forwarding  all  proposed  changes  to  the  appropriate  (Lead  Nation/CJFHQ)  Information 
Manager  (IM). 


A-8 


APPENDIX  B 


Organizational  Structure  Characterizations 


Organizational  Structure 
Traditional :  Vignette  C  (UK  Lead) 


Figure  B-la: 

Organizational 
Structure  Vignette  C, 
Traditional 

During  this  vignette  the 
‘lead’  country  built  the 
MA  and  COA  while  the 
contributing  countries 
provided  input  via  text 
chat.  The  UK  approach 
was  to  combine 
development  and  review 
process.  While  they  were 
building,  the  other  nations 
were  reviewing. 


Organizational  Structure 
Integrated :  Vignette  C  (US  Lead) 
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Figure  B-lb: 
Organizational 
Structure  Vignette  C, 
Traditional 

The  US  delegated  COA 
development  lead  to  UK. 
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Organizational  Structure 
Traditional :  Vignette  E  (GE  Lead) 


Figure  B-2a: 
Organizational 
Structure  Vignette  E, 
Traditional 


The  Lead  Nation  had 
primary  responsibility  for 
MA  development,  but  all 
nations  were  involved  in 
a  brainstorming  session 
prior  to  the  start  of  MA 
development.  The  Lead 
Nation  also  had  primary 
responsibility  for  COA 
development,  but 
monitored  the 
discussions  of  other 
nations  during  the 
development. 


Organizational  Structure 
Integrated :  Vignette  E  (US  Lead) 
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Figure  B-2b: 
Organizational 
Structure  Vignette  E, 
Integrated 


The  US  delegated  COA 
development  lead  to 
AUS. 


B-2 


Organizational  Structure 
Traditional :  Vignette  F  (UK  Lead) 


Figure  B-3a: 
Organizational 
Structure  Vignette  F, 
Traditional 


During  COA  development 
all  nations  were  requested 
to  supply  comments  and 
suggestions  on  the  force 
structure  required. 


Organizational  Structure 
Integrated :  Vignette  F  (US  Lead) 
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Figure  B-3b: 
Organizational 
Structure  Vignette  F, 
Integrated 


US  COA  as  well.  Those 
not  involved  in  MA 
development  were 
directed  to  take  part  in  a 
discussion  on  a  broad 
planning  concept  for  the 
COA. 
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Organizational  Structure 
Traditional :  Vignette  G  (GE  Lead) 


Figure  B-4a: 
Organizational 
Structure  Vignette  G, 
Traditional 


During  COA  development 
each  nation  had  a  specific 
Intel  task:  US  to  assess 
opponent’s  military 
capability,  UK  to  assess 
political/economic 
situation,  and  AUS  had 
the  EJC2E  task. 


Organizational  Structure 
Integrated :  Vignette  G  (Ail  Lead) 
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Figure  B-4a: 
Organizational 
Structure  Vignette  G, 
Integrated 


Even  though  there  were 
teams  and  all  hands 
review  there  were  some 
complaints  that  the  lead 
nation  appeared  to  be 
working  off-line. 
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Organizational  Structure 
Traditional :  Vignette  H  (US  Lead) 
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Figure  B-5a: 
Organizational 
Structure  Vignette  G, 
Traditional 


Organizational  Structure 
Integrated :  Vignette  H  (GE  Lead) 
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Figure  B-5b: 
Organizational 
Structure  Vignette  H, 
Integrated 


During  this  vignette  the 
‘lead’  country  built  the 
MA  and  COA  while  the 
contributing  countries 
provided  input  via  text 
chat.  Functional  teams 
were  not  used. 
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Organizational  Structure 
Traditional :  Vignette  I  (US  Lead) 
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Figure  B-6a: 
Organizational 
Structure  Vignette  I, 
Traditional 


Organizational  Structure 
Integrated :  Vignette  I  (UK  Lead) 
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Figure  B-6b: 

Organizational 
Structure  Vignette  I, 
Integrated 

During  this  vignette  the 
‘lead’  country  built  the 
MA  and  COA  while  the 
contributing  countries 
provided  input  via  text 
chat.  Functional  teams 
were  used  for  logistics 
and  Intel. 

UK  style  was  to 
combine  development 
and  review  process. 
While  they  were 
building,  the  other 
nations  were  reviewing. 
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Organizational  Structure 
Traditional :  Vignette  J  (AU  Lead) 
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Figure  B-7a: 
Organizational 
Structure  Vignette  J, 
Integrated 

Red  cross-hatch 
markings  indicate  that 
Germany  lost  its 
network  connection  for 
the  entire  vignette. 


LEAD 


LEAD 


Organizational  Structure 
Integrated :  Vignette  J  (US  Lead) 
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Figure  B-7b: 
Organizational 
Structure  Vignette  J, 
Integrated 

Red  cross-hatch  markings 
indicate  that  Germany  lost 
its  network  connection  for 
the  entire  vignette. 
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APPENDIX  C 


Vignette  Evaluation  Questionnaire  Data 


The  three  questions  shown  in  Table  C-l  were  administered  in  the  Vignette  Questionnaires,  under 
Section  II,  Process  Characterization.  These  questions  were  distributed  during  Vignettes  C 
through  J.  The  table  shows  the  numerical  breakdown  of  responses  by  vignette  for  the  Integrated 
Team. 


Table  C-l 

Vignette  Questionnaire  Section  II,  Process  Characterization,  Integrated  Team 


1 .  Lack  of  continuous  audio  was  a  significant  detriment  to 
collaboration 

Agree 

Undecided 

Disagree 

No  Opinion 

Vignette 

ID 

Total 

Responses 

Integrated 

1 

0 

1 

0 

C 

2 

Integrated 

1 

1 

3 

1 

E 

6 

Integrated 

1 

0 

2 

1 

F 

4 

Integrated 

3 

1 

1 

0 

G 

5 

Integrated 

6 

2 

2 

0 

H 

10 

Integrated 

5 

0 

1 

0 

1 

6 

Integrated 

3 

0 

3 

1 

J 

7 

20 

4 

13 

3 

40 

2.  More  input  from  coalition  partners  during  initial  development 
of  the  Mission  Analysis  or  COA  would  improve  the  product. 

2 

Integrated 

1  1  0  1  1 

0 

C 

Integrated 

3  i  2  i-  1 

0 

E 

6 

Integrated 

1  r  °  T 3 

0 

F 

4 

Integrated 

1  1  r  3 

0 

G 

5 

Integrated 

6  \  1  r  3 

0 

H 

10 

Integrated 

2  i  1  r  2 

1 

1 

6 

Integrated 

2  1  2  1  3 

0 

J 

7 

16  |  7  I  16 

1 

40 

3.  During  collaborative  planning  I  prefer  a  predefined  structure 
(such  as  a  Mission  Analysis  matrix)  to  a  free-fbrm  notepad  tool. 

Integrated 

2  1  0  1  0 

0 

C 

2 

Integrated 

6  T  0  |  1 

0 

E 

7 

Integrated 

3  \  0  \  1 

0 

F 

4 

Integrated 

2  [  1  2 

0 

G 

5 

Integrated 

6  \  2  \  2 

0 

H 

10 

Integrated 

4  f  1  _ |  1 

0 

1 

6 

Integrated 

4  °  T~  3 

0 

J 

7 

27  |  4  |  10 

0 

41 

C-l 


The  three  questions  shown  in  Table  C-2  are  the  same  as  those  shown  in  Table  C-l.  This  table 
shows  the  numerical  breakdown  of  responses  by  vignette  for  the  Traditional  Team. 


Table  C-2 


Vignette  Questionnaire  Section  II,  Process  Characterization,  Traditional  Team 


1 .  Lack  of  continuous  audio  was  a  significant  detriment  to 
collaboration 

Agree 

Undecided 

Disagree 

No  Opinion 

Vignette 

ID 

Total 

Responses 

Traditional 

0 

0 

2 

0 

C 

2 

Traditional 

2 

0 

12 

1 

E 

15 

Traditional 

0 

0 

1 

0 

F 

1 

Traditional 

0 

7 

4 

1 

G 

12 

Traditional 

1 

3 

7 

2 

H 

13 

Traditional 

0 

1 

6 

1 

1 

8 

Traditional 

0 

1 

4 

2 

J 

7 

3 

12 

36 

? 

58 

2.  More  input  from  coalition  partners  during  initial  development 
of  the  Mission  Analysis  or  COA  would  improve  the  product. 

Traditional 

0 

2 

0 

0 

C 

2 

Traditional 

8 

2 

5 

0 

E 

15 

Traditional 

1 

1 

0 

0 

F 

2 

Traditional 

4 

6 

2 

0 

G 

12 

Traditional 

5 

3 

5 

0 

H 

13 

Traditional 

3 

1 

6 

0 

1 

10 

Traditional 

2 

0 

4 

0 

J 

6 

23 

15 

22 

0 

60 

3.  During  collaborative  planning  I  prefer  a  predefined  structure 
(such  as  a  Mission  Analysis  matrix)  to  a  free  form  notepad  tool. 

Traditional 

2 

0 

0 

0 

C 

2 

Traditional 

9 

2 

1 

2 

E 

14 

Traditional 

1 

0 

0 

0 

F 

1 

Traditional 

9 

2 

1 

0 

G 

12 

Traditional 

6 

4 

2 

1 

H 

13 

Traditional 

7 

1 

2 

0 

1 

10 

Traditional 

4 

0 

2 

0 

J 

6 

38 

9 

8 

3 

58 

Figures  C-l  through  C-3  present  the  above  tabulated  data  in  graphical  fonn.  Figures  C-4  through 
C-6  break  out  the  same  information  for  each  question  by  experiment  week.  Figures  C-l  through 
C-9  break  out  the  information  by  participating  country. 


C-2 


Figure  C-l:  Process 
Characterization 
Question  #1 


The  Integrated  Team 
submitted  40  responses  to 
Question  1  and  the 
Traditional  Team 
submitted  58  responses. 


2.  More  Input  from  Coalition  Partners  During  The 
Mission  Analysis  Or  COA  Would  Improve  The  Product 
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Figure  C-2:  Process 
Characterization 
Question  2 


The  Integrated  T earn 
submitted  40 
responses  to  Question 
2  and  the  Traditional 
Team  submitted  60 
responses. 
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Figure  C-3:  Process 
Characterization 
Question  #3 


The  Integrated  T earn 
submitted  4 1  responses  to 
Question  3  and  the 
Traditional  Team 
submitted  58  responses. 


Vignettes  C  Through  F  (Week  1) 


1.  Lack  of  continuous  audio  was  a  significant 
detriment  to  collaboration. 


Agree  Undecided  Disagree  No  Opinion 


Vignettes  G  Through  J  (Week  2) 


Agree  Undecided  Disagree  No  Opinion 


Figure  C-4:  Comparison 
Of  Process 
Characterization 
Question  1  By  Experiment 
Week 


The  graphs  show  a 
comparison  of  the  responses 
given  during  the  first  and 
second  weeks  of  the 
experiment.  Values  plotted 
are  the  percent  of  either  total 
Integrated  or  total 
Traditional  responses 
returned  for  this  question 
each  week. 
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Vignettes  C  Through  F  (Week  1) 


Agree  Undecided  Disagree  No  Opinion 


2.  More  input  from  coalition  partners  during 
initial  development  of  the  Mission  Analysis  or 
COA  would  improve  the  product. 


Vignettes  G  Through  J  (Week  2) 


Agree  Undecided  Disagree  No  Opinion 


Figure  C-5:  Comparison  Of 
Process  Characterization 
Question  2  By  Experiment 
Week 


% 


Vignettes  C  Through  F  (Week  1) 


3.  During  collaborative  planning  I  prefer  a 
predefined  Structure  (such  as  a  Mission  Analysis  matrix) 
to  a  free-form  notepad  tool. 


Vignettes  G  Through  J  (Week  2) 


Figure  C-6:  Comparison 
Of  Process 
Characterization 
Question  3  By  Experiment 
Week 
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Figure  C-7:  Process 
Characterization 
Question  1,  Breakdown 
by  Country 


These  graphs  provide  a 
breakdown  of  the  number 
of  responses  by  team  and 
country  for  each  of  the 
three  process 

characterization  questions. 
Data  include  Vignettes  C 


Figure  C-8:  Process 
Characterization 
Question  2,  Breakdown 
by  Country 


Data  include  Vignettes  C 
through  J. 
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Figure  C-9:  Process 
Characterization 

Question  3  By  Country 


Data  include  Vignettes  C 
through  J. 


The  following  figures  present  the  data  collected  from  the  Vignette  Questionnaire,  Section  III, 
LOE  Evaluation  of  Tools. 
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Traditional  Team 


Autonomy  supported  the  planning 
process  in  a  satisfactory  manner. 

Agree 

Undecided 

Disagree  |  N/A 

Team 

Total 

10 

12 

12  28 

Traditional 

62 

7 

11 

14  22 

Integrated 

54 

Figure  C-10:  Vignette  Questionnaire  Section  III,  LOE  Evaluation  of  Tools 

During  a  post-experiment  meeting  at  APL  to  discuss  preliminary  results,  some  of  the  participants  who  had  been 
observers  expressed  surprise  that  there  were  any  agree  responses.  Questionnaires  were  reviewed  again  and  the 
following  breakdown  of  Agree  responses  was  observed: 

Traditional  Team 

•70%  of  the  Agree  responses  were  from  Germany. 

•20%  of  the  Agree  responses  were  from  the  United  Kingdom. 

•10%  of  the  Agree  responses  were  from  Australia. 

•Responses  came  from  Vignettes  F,  G  and  H. 

Integrated  Team 

•86%  of  the  responses  came  from  the  US  team.  The  majority  of  the  Agree  responses  came  from  Vignettes  G 
through  I,  and  all  were  from  the  US  for  these  vignettes. 

•At  least  3  of  the  US  Agree  responses  came  from  the  Integrated  Teams  Intel  person. 

•The  other  3  US  Agree  responses  may  be  invalid.  It  appears  that  another  Integrated  player  may  have  duplicated 
the  Intel  persons  questionnaire  to  save  time.  While  this  person  appears  to  have  then  edited  the  duplicated  form 
in  some  areas,  the  tools  responses  appear  to  be  identical  during  several  vignettes.  It  could  also  be  that  he 
happened  to  agree  and  felt  no  need  for  change. 

Because  there  appeared  to  be  more  than  one  instance  of  Questionnaire  duplication  during  this  experiment 
(duplication  did  not  appear  limited  to  the  US)  some  of  the  tools  data  should  be  regarded  with  caution.  If  a  form 
is  duplicated  because  the  participant  agreed  with  the  responses  of  another  participant  and  wanted  to  save  time  it 
is  perfectly  legitimate,  however  if  the  participant  forgot  to  check  all  responses  to  make  sure  they  agreed  with  his 
own  opinions  the  form  effectively  becomes  a  double  vote  for  the  original  respondent. 
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Traditional  Team 


□  Agree 

□  Undecided 

□  Disagree 

□  N/A 


Integrated  Team 
18% 


Brain  EKP  supported  the  planning 
process  in  a  satisfactory  manner. 

Agree 

Undecided 

Disagree| 

N/A 

Team 

Total 

19 

16 

12 

17 

Traditional 

64 

10 

14 

9 

22 

Integrated 

55 

Figure  C-ll:  Vignette  Questionnaire  Section  III,  LOE  Evaluation  of  Tools 


The  breakdown  of  Agree  responses  for  the  BrainEKP  question  were  as  follows: 

Traditional  Team  (29%  of  Total  Traditional  Respondents  Agreed) 

•  58%  of  the  Agree  responses  were  from  Australia 

•  21%  were  from  the  United  Kingdom 

•  16%  were  from  the  US 

•  5%  were  from  Germany 

•  68%  of  all  Agree  Responses  were  provided  during  the  second  week  of  the  experiment  (Vignettes  G-J) 


Integrated  Team  (18%  of  Total  Integrated  Respondents  Agreed) 

•  70%  were  from  the  US 

•  30%  were  from  GE 

•  60%  of  the  Agree  Responses  were  provided  during  the  second  week  of  the  experiment  (Vignettes  G-J). 
83%  of  those  were  from  the  US. 
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Traditional  Team 


14% 


Integrated  Team 


□  Agree 

■  Undecided 

□  Disagree 

□  N/A 


21% 


Figure  C-12:  Vignette 
Questionnaire  Section  III, 
LOE  Evaluation  of  Tools 


Due  to  system  (Groove)  complexity,  1 
tended  to  use  only  a  few  functions. 

Agree 

Undecided 

Disagree 

N/A 

Team 

Total 

7 

12 

35 

4 

Integrated 

58 

12 

9 

41 

2 

Traditional 

64 

Groove  seems  to  disrupt  the  way  1 
normally  like  to  arrange  my  work. 

Agree 

Undecided 

Disagree 

N/A 

Teams 

Total 

10 

16 

37 

0 

Traditional 

63 

4 

18 

32 

4 

Integrated 

58 

Figure  C-13:  Vignette 
Questionnaire  Section  III, 
LOE  Evaluation  of  Tools 
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Traditional  Team 


Integrated  Team 
o% 

32 

■  Agree 

■  Undecided 

□  Disagree 

□  N/A 


Groove  supported  the  planning 
process  in  a  satisfactory  manner. 

Agree 

Undecided 

Disagree 

N/A 

Team 

Total 

50 

7 

7 

0 

Traditional 

64 

28 

12 

19 

0 

Integrated 

59 

Figure  C-14:  Vignette 
Questionnaire  Section  III, 
LOE  Evaluation  of  Tools 


Traditional  Team 


81% 


□  Agree 

El  Undecided 

□  Disagree 

□  N/A 


Integrated  Team 


88% 


Figure  C-15:  Vignette 
Questionnaire  Section  III, 
LOE  Evaluation  of  Tools 


1  can  understand  and  act  on  the 
information  provided  by  Groove. 

Agree 

Undecided 

Disagree 

N/A 

Teams 

Total 

52 

3 

7 

2 

Traditional 

64 

51 

6 

1 

0 

Integrated 

58 

C-ll 


Traditional  Team 


Figure  C-16:  Vignette 
Questionnaire  Section  III, 
LOE  Evaluation  of  Tools 


1  would  like  to  use  Groove  in  a  real  operation. 

Agree  |  Undecided  |  Disagree  |  N/A 

Team 

Total 

37  17  91 

Traditional 

64 

36  14  71 

Integrated 

58 

C-12 


The  following  discussion  presents  the  summary  notes  and  supporting  comments  from  Section  II 
of  the  Vignette  Evaluation  Questionnaire. 


Traditional  Team: 


Section  II  : Process 

Collaboration/Process  -  Traditional 

•  Leadership  was  repeatedly  cited  as  a  critical  factor  in  the  process.  Lead 
Nation  determined  the  process  and  level  of  collaboration. 

•  Various  doctrinal  (possibly  cultural)  issues  were  cited. 

•  Text  chat  was  definitely  preferred  over  audio. 

•  Participants  had  mixed  reactions  to  (amount  of)  data  flow. 

•  As  LOE  progressed,  team  developed  Standard  Operating  Procedures  that 
took  advantage  of  the  collaboration  tool;  process  became  more  integrated. 

Overall,  team  players  indicated  that  the  tool  enabled  the  ‘Traditional’  process 
to  be  conducted  in  a  more  collaborative  manner. 
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Leadership: 

(LN)  lead  was  a  strong  one”  [E] 

•  “During  the  traditional  planning  process  the  lead  team  needs  time  alone  for  discussion  as  I  imagine  that  the  lead 
team  is  responsible  for  the  plan...  there  are  time  blocks  where  the  lead  team  needs  to  work  on  its  own...  the  lead 
team  rules  and  has  to  decide  somewhere”  [E] 

•  “If  the  lead  team  does  not  listen  to  input,  what  is  the  point  in  providing  it.”  [E] 

•  “Planning  teams  possessed  lots  of  experience  which  was  not  utilized  by  the  Lead  Nation”  [E] 

•  (Lead  Nation)  “...didn't  involve  others  actively  in  their  planning  process,  own  group  seemed  to  be  kind  observer”  - 

[FI 

•  “...  the  (LN)  went  into  a  nation  huddle  for  COA  development,  but  that  tends  to  be  the  way  of  the  traditional 
planning  process...”  [H] 

•  “Planning  activity  worked  better  today.  (LN)  team  involved  the  other  nations  in  their  MA  and  COA 
development.” 

•  “(LN)  again  made  good  use  of  other  nations’  planning  teams  during  COA  dev.’  [I] 

•  “Lead  tried  to  give  out  too  much  of  it’s  own  responsibility  to  coalition  partners.  “  [I] 

•  “Would  help  if  the  leader  were  to  lead...  Abrogating  responsibility  for  all  the  effort  does  not  lead  to  team  spirit.  “  [I] 

•  “Leaders  lead,  supporting  nations  contribute”  [I] 

Doctrinal/Cultural  Issues: 

•“...  all  teams  had  different  understandings  on  the  necessary  level  of  detail  in  the  planning  process”  [E] 

•  “...plan  was  so  lacking  in  detail  -  the  result  is  the  problems  would  have  surfaced  later.”  [E] 
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•  “  And  different  countries,  different  opinions  about  planning. . [E], 

•  “Seemed  that  different  opinions  of  the  sense  of  a  COA  are  within  the  different  groups:  COA  just  a  step  toward  the 
plan."  [F] 

•  “..in  some  instances  though,  it  was  a  bit  overwhelming  to  the  point  that  it  appeared  the  UK  was  actually  building  a 
timeline  to  commence  operations.’  [F] 

•  “Some  differences  over  command  structures  persists”  [FI] 

•  “Review  doctrine  on  certain  points  and  insure  multinational  agreement  prior  to  ops”  [FI] 

•  “  All  lead  nations  continue  to  do  planning  in  a  different  manner.  This  caused  confusion  from  time  to  time.”  [FI] 

•  “Cultural  differences  not  always  a  positive  experience”  [G] 

•  “...  think  they  know  it  all  and  try  to  take  over  from  lead  nation”.  [G] 

•  “..(LN)  managed  to  resist  the  tendency  to  make  this  operation  more  than  it  is.”  [FI]  -  tendency  for  heavy  use  of 
forces  was  cited  several  times 

•  “...  (LN)  have  got  to  listen  to  advice  and  not  storm  off  with  a  CVBG’  [FI] 


Audio: 

•“Use  of  chat  board,  not  voice,  made  it  easier  to  record  and  develop  solution.”  [E] 

•  “Discussion  was  much  more  open  and  free  flowing  in  the  text  chat.  “  [E] 

•  “VOICE  BARELY  USED  AS  LESS  EFFICIENT  IN  THIS  CASE”  [F] 

•  “The  use  of  chat  not  voice  was  a  positive  contribution  to  the  maintenance  of  the  aim”  [F] 


Data  Flow/  Situational  Awareness: 

•“Too  many  info  for  the  small  planning  group”  [E] 

•  “You  cannot  always  chit  and  chat  and  discuss  with  the  tools  at  any  time. . .”  [E] 

•  “To  many  spurious  in  puts  from  other  nations.  No  attempt  to  produce  planning  teams  outside  nation  groups.”  [G] 

•  “Worked  well  except  for  ‘clap-trap’  going  on  during  quiet  periods.” 

•  “Continuous  data  feed  provided  by  GROOVE  from  unrecognizable  sources  seriously  hampered  the  intended  data 
flow.”  [H] 

•  “  The  immense  amount  of  information  given  by  so  many  experts  can  hamper  own  consideration  flow  once  in  a 
while.  Too  many  opinions  are  not  always  helpful”  [H] 


SOP: 


•Traditional  process  evolved:  “Difficult  to  use  if  the  traditional  planning  process  will  be  modified  by  lead  nation 
without  prior  coordination  with  other  nations. . .  ’  [F] 

•  “Teamwork  is  improving.  We’re  developing  a  common  language  ‘  [F] 

•  “Less  arguing  between  planning  cells  concerning  most  effective  use  of  each  other's  assets.”  [G] 

•  “Traditional  planning  cells  developed  an  informal  Standard  Operating  Procedure  (SOP)  that  was  developed  as 
traditional  planning  cells  cycled  through  the  vignettes.’  [H] 

•  “...we  are  beginning  to  develop  standard  operating  procedures  and  planning  templates  that  take  advantage  of  the 
collaborative  planning  tool” 

•  “Development  of  a  single  planning  “stovepipe”  instead  of  one  for  each  national  planning  cell  participating  in  the 
LOE.  This  has  cut  down  on  time  required  to  develop  conduct  MA  and  COA.”  [H] 

•  “  The  Lead  Nation  appointed  Nations  to  Lead  Components  assigned  tasks  and  requested  COAs.’  [J] 

•  “US  again  made  good  use  of  other  nations’  planning  teams  during  COA  dev”  [I]. 

•  “Coordination  and  collaboration  seemed  to  work  really  well  today.”  [J] 

•  “  Ability  to  coordinate  directly  sped  the  MA  and  COA  development.  Cooperative  attitude  and  awareness  of  other 
teams  capabilities  increased  the  speed  of  development”  [J] 
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Integrated: 


Section  II  :Process 

Collaboration/Process  -  Traditional 

•  Learning  curve  evident  since  process  was  new  to  non-US  teams. 
Participants  noted  that  they  became  more  efficient  with  more  experience. 
Recommendations  provided. 

•  Though  some  participants  preferred  chat  text,  audio  was  a  significant  part 
of  the  process  for  the  duration  of  the  LOE. 

•  Continuous  situational  awareness  (of  team  activity)  considered  critical  to 
effectiveness  of  process. 

•  Some  leadership  and  cultural  issues  were  raised,  though  overall 
appeared  to  be  less  of  an  issue  than  for  Traditional  team  . 

•  Process  stressed  capabilities  of  network  and  collaboration  tool. 
Collaboration  tool  problems  viewed  as  significant  detriment  to  planning 
process. 

Overall,  participants  were  positive  about  the  Integrated  planning  process. 
Were  able  to  persevere  in  spite  of  technical  problems. 
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Learning  Curve: 

•“  Improved  fluency  with  the  system  -  there  is  a  clear  learning  curve”  [E] 

•  “I  have  seen  a  UK  led  and  now  have  stumbled  through  an  Aus  one.  However,  both  were  muddled  and  would  have 
benefited  from  actually  seeing  how  it  is  done  rather  than  learning  it  by  making  mistakes.”  [E] 

•  “Increasing  team  rapport  and  familiarity  with  processes  evident”  [E] 

•  “The  US  led  COA  development  was  good  to  see.  It  appears  that  AUS  and  the  UK’s  previous  attempts  were  in  fact 
significantly  better  done  than  I  thought  at  the  time.  US  took  many  of  our  ideas  and  repeated  them.  Good  confidence 
builder  for  future  vignettes.”  [F] 

•  “There  has  been  a  steady  increase  in  participation  levels  by  all  nations  as  personnel  have  become  more  familiar 
with  the  software  and  the  procedures.”  [F] 

•  “...  even  after  x  number  of  exercises,  the  planning  process  is  still  a  little  mysterious.”  [G] 

RECOMMENDATIONS: 

•“Need  to  develop  business  rule  to  alert  members  to  communications  failures  before  too  much  information  fails  to 
pass  and  SA  is  affected.”  [G] 

•  “  Create  business  rules  to  drive  default  to  secondary  means  of  communication  when  team  is  less  than  100% 
capable  of  using  primary  means.”  [G] 

•  “closer  advisory/business  roles  recommended”  [G] 

•  “Info  management  need  closer  advisory’ [E] 

•  “Suggest  we  have  a  formal  step  agreeing  ConOps  and  COA  schematic  prior  to  moving  on  to  discuss  remainder  of 
COA  template”  [G] 
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•  Fixed  Agenda:  “The  tempo  of  this  vignette  was  too  slow.  There  was  too  much  time  allocated  to  each  step  of  the 
process,  leading  to  extended  periods  of  inaction.  It  was  evident  that  extended  time  to  develop  MA  and  COA  does 
not  produce  better  products.  “  [F] 

•  “Moved  too  slowly.  Given  this  is  an  experiment,  need  to  avoid  being  unduly  wedded  to  initial  process.”  [F] 

Audio: 

•“Written  chat  leads  to  more  accuracy,  discipline  and  shortness”  [E] 

•  “  Audio  (communications)  is  unreliable  and  should  be  minimized  to  commands  and  business  roles”  [E] 

•  “Intermittent  audio  continues  to  be  a  detriment  to  the  free-flow  of  knowledge  and  idea  sharing  that  forms  the  basis 
of  effective  collaboration.  “  [E] 

•  “Utilize  audio  as  the  primary  tool.  Use  chat  as  the  backup. ”[G] 

•  Potential  Improvements:  “Greater  use  of  audio  to  discuss  conflicts  of  opinion  or  confusions  with  assets  etc”  [H] 

•  “The  audio  allowed  me  to  control  the  process  effectively  -  without  it  it  would  have  been  difficult  to  interpret  other 
nations  intent.”  [I] 

Situational  Awareness: 

•“...not  sure  about  effectiveness  of  team  members  working  in  separate  places  (loss  of  SA  while  working  in  other 
spaces)  [C] 

•  “Need  good  comms/  good  for  situational  awareness/ensures  ownership  of  the  product  throughout/Need  to  maintain 
consensus  but  easier  to  achieve  in  this  process  .”[E] 

•  “Need  to  develop  business  rule  to  alert  members  to  communications  failures  before  too  much  information  fails  to 
pass  and  SA  is  affected.”  [F] 

•  “Situational  awareness  remained  high  throughout”.  [F] 

•  “Situational  awareness  lost  upon  occasion  when  team  members  employed  a  tool  they  did  not  know  (and  had  no 
indication  of)  was  not  fully  functioning  or  had  failed  completely.  “[F] 

•  “Lead  stated  that  they  were  going  off  line  to  smooth  up  the  MA.  Should  have  stayed  in  collaboration  to  make 
changes,  unless  just  grammar.  Communications  was  lacking: ’[G] 

•  “Situational  awareness  allowed  rapid  changes  of  lead”  [G] 

•  “  Good  SA  and  core  teamwork  while  comms  good.  When  UK  needed  to  take  lead  we  were  able  to  do  so  because 
of  the  high  degree  of  collaboration  up  until  technical  probs”  [H] 

•  “Integrated  planning  relies  on  saturation  of  all  communications  media  for  parallel  and  concurrent  efforts’  [H,I] 


Leadership: 


•“Poor  delegation  of  tasks  within  the  lead  team  meant  that  Intel  lead  was  overtasked  (Intel,  SF  and  lead  for  special 
COA).  The  main  COA  was  delegated  away  from  the  lead.  “  [J] 

•  “Lead  stated  that  they  were  going  off  line  to  smooth  up  the  MA.  Should  have  stayed  in  collaboration  to  make 
changes’  [G] 

•  “Lead  nations  are  focusing  on  managing  collaboration  rather  than  influencing  the  product...”  [J] 


Tool  Problems: 

•“Tool  works  when  there  is  a  small  number  of  players.”  [C] 

•  “Any  degradation  in  communications  is  bound  to  adversely  impact  a  process  based  on  free  collaboration.  “  [G] 

•  “Vastly  improved  communications  allowed  greater  depth  of  collaboration. . .”  [E] 
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•  “Process  is  highly  dependent  upon  SOME  form  of  communications  with  the  ability  to  exchange  information  in 
real  time.  Stumbled  badly  at  first  when  audio  went.  Unclear  to  lead  planner  when  to  rely  upon  fallback  method. ’[G] 

•  “Degradation  of  comms  reduced  ability  to  plan  interactively”  [H] 

•  “Understand  tool  is  not  the  evaluation  here,  but  without  it  the  collaboration  breaks  down  and  momentum  is  lost. 
The  planning  then  turns  into  a  one  nation  (lead)  effort”  [H] 

•  “  When  defaulting  to  the  “less  than  optimum  tool  suite”,  the  process  becomes  highly  traditional  in  nature,  the 
whole  process  defaulting  to  whoever  happens  to  be  lead  and  the  lead  defaulting  to  whoever  can  make  inputs  on  the 
tool.”  [H] 

•  “Can’t  think  of  any  way  to  de-couple  the  success  of  the  Integrated  Process  with  the  success  of  an  integrating  tool. 
Find  a  tool  that  works — consistently.”  [I] 


Overall  Satisfaction: 

•“Continued  use  shows  that  the  process  is  viable  “  [C] 

•  “Unfortunately,  network  problems  precluded  our  involvement  in  approx..  50%  of  the  vignette.  Nevertheless,  once 
again  the  collaborative  planning  process  using  groove  proved  an  effective  planning  system.  “  [H] 

•  “Staffs  continue  to  plan  despite  tool  failure  or  degradation.”  [H,I] 

•  Strengths:  “teamwork. .  .cooperation” 

•  “The  International  team  has  become  proficient. . .”  [I] 

•  “We  have  all  gotten  more  efficient”  [I] 

•  “The  process  takes  longer  to  draft  but  reduces  time  to  agree  final  draft  at  the  end  “  [I] 

•  “All  inputs  from  each  nation  was  considered  and  put  into  the  MA  or  COA  as  appropriate.”  [I] 

•  “Today’s  planning  worked  very  well.  Information  transfer  worked  very  well  and  all  planners  were  able  to 
contribute  positively.  Audio  supplemented  text  chat  very  well.  Network  performed  superbly  and  supported 
vignette.”  [J] 

•  “...  as  a  distributed  team  we  seem  content  with  the  way  of  working”  [J] 
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APPENDIX  D 


Team  Working  and  Planning  in  Coalition  Distributed  Teams  Survey 


Johns  Hopkins  University 
Applied  Physics  Laboratory 

Team  Working  and  Planning  in 
Coalition  Distributed  Teams  Survey 

•  Developed  by  UK;  UK  is  primary  lead  for  analyzing 
survey  data.  The  UK  analysis  effort  focuses  on 
human  factors. 

•  APL  team  generated  summary  results  in  order  to 
correlate  survey  inputs  with  other  process-related 
data  collected  during  MN  LOE  I. 

•  Supporting  comments  are  provided  in  Notes  sections. 


MN  LOE  I  Analysis  Workshop  1/9-1/10 
-  For  workshop  use  only  - 


Johns  Hopkins  University 
Applied  Physics  Laboratory 

Team  Working  and  Planning  in 
Coalition  Distributed  Teams  Survey 

•  Survey  questions  provided  a  scale  of  1-7,  spanning  Strongly 
Disagree  (1)  to  Neutral  (4)  to  Strongly  Agree  (7). 

•  In  the  following  slides,  three  major  categories  are  reported: 

•  1-3=  Disagree 

•  4  =  Neutral 

•  5-7  =  Agree 

•  Charts  report  Traditional  and  Integrated  team  results  separately 
( total  #  of  completed  surveys  per  team  is  provided  ). 

•  One  participant  who  had  worked  both  processes  was  included  in 
the  ‘  Traditional’  sample. 

MN  LOE  I  Analysis  Workshop  1/9-1/10 
-  For  workshop  use  only  - 
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Johns  Hopkins  University 
Applied  Physics  Laboratory 


Collaboration  Survey  -  Interoperability 


National  military  doctrine  did  not  differ  from  the  doctrine  of  other  partner 
nations  sufficiently  to  force  an  adaptation  of  usual  working  practices. 


Question  22 


T  (16) 


1(11) 


✓ 

■ 

/ / 

0%  50%  1 00% 

%cf  Responses 


□Disagree 

■Neutial 

□Agree 


MN  LOE I  Analysis  Workshop  1/9-1/10 
-  For  workshop  use  only  - 


Question  2.2 

Traditional 

•  “(1)  Center  of  Gravity  Constructs  (AU  Army  doctrine  differs  slightly  from  AU  “Joint” 
doctrine  but  reflects  UK,  GE  and  US  Joint  doctrine  -  apparently)  (2)  Format  and  use  of  mission 
statements  (foreign  tendency  is  to  spell  out  a  mission  as  a  story  of  bullet  points  which  reads  to 
an  Australian  like  a  line-of-operations  or  a  method  statement)”. 

•  “THIS  WAS  NOT  A  PROBLEM  -  EXCEPT  FOR  VERY  MINOR  ISSUES” 

•  “All  countries  had  a  variation  of  the  same  doctrine. . .” 

•  “US  Doctrine  was  clearly  at  odds  with  NATO  Doctrine  which  US/AUS/GE/UK  are  supposed 
to  use” 

•  “Other  Nations  seemed  to  want  to  leap  into  detailed  planning  rather  than  developing  generic 
COAs  to  compare  and  then  select  for  detailed  planning.” 

•  “...  German  planning  process  was  more  in  accordance  with  the  planning  level  ( Operational 
Level,  CJTF)  than  the  others,  where  the  planning  was  more  on  the  tactical  level'  (This 
participant  actually  worked  in  both  processes) 

Integrated 

•  “  OVERALL  THIS  HAD  BEEN  NOT  A  PROBLEM  -  EXCEPT  FOR  SOME  MINOR 
ISSUES;  THERE  HAD  BEEN  SOME  DIFFERENCES  BETWEEN  GERMAN  PLANNING 
PROCEDURE  AND  NATO  GOP.” 

•  “Issues  centered  around  rules  of  engagement  and  military  presence.  Mission  analysis  was 
similar  except  for  the  way  we  think  about  centers  of  gravity.” 
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Johns  Hopkins  University 
Applied  Physics  Laboratory 

Collaboration  Survey  -  Interoperability 


The  national  cell  was  very  effective  in  its  decision  making  and  procedures 


MN  LOE I  Analysis  Workshop  1/9-1/10 
-  For  workshop  use  only  - 


Question  2.3 

There  was  some  confusion  among  the  participants  as  to  the  meaning  of  the  terms  ‘national’  and 
‘coalition’  cells.  Survey  definitions: 

National  cell:  The  national  Group  you  worked  with  in  this  experiment 

Coalition  cell:  The  international  team  you  worked  with  made  up  of  geographically  isolated  cells 
in  the  UK,  USA,  Germany  and  Australia> 

Traditional 

•  “The  team  worked  very  well  together  and  had  a  broad  range  of  operational  experience.” 

•  “SOMETIMES  MEMBERS  CONCERRED  WITH  DECISIONS  JUST  TO  GET  THE  COA 
DONE  IN  TIME.” 

•  “At  the  national  level  we  were  all  working  to  the  same  doctrine,  had  the  benefit  of  working 
physically  alongside  each  other  and  could  resolve  differences  quickly” 

•  “Due  to  manpower,  system  handling  problems  and  different  opinion  concerning  the  planning 
level  (see  nr.2  above)  decision  making  and  procedures  were  hampered  considerably” 

•  “We  did  not  have  a  proper  mix  of  personnel.”  [Participant  responded  “5”.] 

Integrated 

•  “The  national  cell  was  not  a  decision  making  authority.  Having  said  that  the  team  was  very 
effective  because  we  were  familiar  with  our  procedures  and  we  are  working  in  the  information 
sharing  end  of  the  planning  process  the  analysis  and  approval  of  a  COA  “ 

•  “ALTHOUGH  WE  DIDN'T  KNOW  US,  DESCISIONS  WERE  MADE  IN  A  HIGHLY 
COMPETENT  AND  QUICK  MANNER!” 

•  “The  process  worked  for  our  cell,  we  were  able  to  blend  skills  and  experience  and  come  up 
with  right  answers” 
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Johns  Hopkins  University 
Applied  Physics  Laboratory 


Collaboration  Survey  -  Interoperability 


Differences  in  culture  [i.e.,  national  attitudes,  behaviours] 
between  nations  forced  the  adaptation  of  working  practices. 


Question  2.4 


□Disagree 

HNeutral 

□Agree 


%  of  Responses 
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-  For  workshop  use  only  - 


Question  2.4 

Traditional 

•  “...preferred  to  work  in  isolation.  This  made  other  nations  appear  redundant.” 

•  “  NO  CHANGE  TO  WORKING  PRACTICES  BUT  CARE  IN  COMMENTS  MADE.  ” 

•  “Some  nations  delegated  too  far. . 

•  “. . .dictatorial  leadership. . .” 

•  “Because  of  different  planning  doctrines  teams  had  to  undergo  a  phase  of  education  in  order  to 
understand  the  different  approaches.” 

•  “My  way  is  the  best  and  only  way  slipped  in  several  times.  “ 

•  “...colloquialisms  and  phrases  to  non  native  English  speakers. . .” 

Integrated 

•  All  ‘Agree’  responses  were  5  (i.e.  slight/moderate  agreement) 

•  “NO  CHANGE  TO  WORKING  PRACTICES. . .” 

•  “...  cultural  tendency  toward  centralization...” 

•  “...  language  differences...” 
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Johns  Hopkins  University 
Applied  Physics  Laboratory 

Collaboration  Survey  -  Interoperability 


The  Coalition  cell  was  very  effective  in  its  decision  making  and 
procedures. 


MN  LOE I  Analysis  Workshop  1/9-1/10 
-  For  workshop  use  only  - 


Question  2.5 

Traditional 

•  “Lead  Nation  concept  worked,  however  the  LOE  did  not  test  national  resolve  or  commitment, 
particularly  in  terms  of  potential  casualties  and  political  issues.” 

•  “...other  nations  were  a  little  tentative  in  their  leadership. . .” 

Integrated 

•  “Other  than  initial  hesitation  to  exercise  a  process  they  were  neither  familiar  with  or 
proponents  of,  I  felt  our  coalition  partners  were  only  slightly  hampered  in  the  speed  of  their 
decisions,  not  the  accuracy  thereof.” 

•  “Considering  all  the  hampering  factors  (see  above)  it  was  understandable  that  the  results  could 
have  been  better.” 
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Collaboration  Survey  -  Interoperability 

Our  national  cell  did  not  have  to  develop  new  ways  of  working  in  order  to 
efficiently  interact  with  the  other  cells  making  up  the  coalition  team. 


MN  LOE I  Analysis  Workshop  1/9-1/10 
-  For  workshop  use  only  - 


Question  2.6 

Traditional 

•  adjusted  daily  to  each  nations  requirements  and  over  the  period  of  the  experiment  an 
‘almost  common’  procedure  was  developed” 

•  “Planning  strategy  had  to  be  changed  during  the  experiment.  The  final  result  was  a 
completely  new  way  to  get  the  work  done.” 

•  “Each  team  had  different  formats  which  each  other  nation  used  and  adapted  to.  This  did  not 
interfere  with  COA  development.  There  were  occasions  when  MA  was  delayed  whilst  nations 
worked  through  different  document  formats.  “ 

•  “WE  ONLY  HAD  TO  ADJUST  IN  MINOR  WAYS:” 

Integrated 

•  Five  of  the  9  responses  were  from  the  US  team,  which  had  the  primary  lead  for  the  Integrated 
process,  had  worked  the  most  with  it  and  each  other  (i.e.,  results  are  skewed  by  US  perspective) 

•  “Our  national  cell  continued  to  adjust  the  agenda,  explore  different  ways  to  assign  the  course 
of  action  tiger  team  and  leverage  expertise  in  our  cell.” 
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Collaboration  Survey  -  Interoperability 


Our  national  cell's  inputs  were  highly  valued  by  the  coalition  team. 


Question  2.7 
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Question  2.7 

Traditional 

•  Most  of  the  neutral  responses  were  due  to  the  fact  that  participants  found  this  hard  to  assess  in 
a  distributed  environment. 

•  “Not  able  to  determine  form  this  end.” 

Integrated 

•  Overall,  positive  response 

•  “THIS  IS  VERY  DIFFICULT  TO  ANSWER!  THE  COMMENTS  OF  EVERY  NATION 
HAD  BEEN  ALWAYS  POSITIVE!  IF  THERE  HAD  BEEN  ANY  REMARKS  OR 
QUESTIONS,  WE  DISCUSSED  THIS  PROBLEM  VIA  CHAT  AND/OR 
COMMUNICATION  and  finally  came  to  a  common  accepted  solution!” 
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Collaboration  Survey  -  Interoperability 


Our  national  cell  did  not  have  serious  disagreements  with  any  final  COAs. 
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Question  2.8 

In  addition  to  the  overall  question,  2.8  also  asked  the  participants  to  rate  each  individual  COA 
from  the  perspective  of  whether  their  national  cell  had  serious  disagreements  with  any  of  the 
final  COAs. 

Traditional 

•  One  participant  responded  “3”  yet  rated  individual  COAs  “6”  -  ?? 

•  “Foreign  temptation  is  to  overkill.  The  situation.  Humanitarian  relief  distracted  them  from 
their  mission  to  maintain  a  standing  ready  force  for  war  and  they  insisted  on  deploying  forces 
that  would  saturate  the  disaster  area  making  more  of  a  burden  than  a  contribution.” 

•  “On  a  number  of  occasions  we  were  not  satisfied  with  the  standard  of  COA  produced.  We 
believed  that  they  mostly  lacked  detail ,  and  were  simply  very  sketchy  outline  plans  with  lists  of 
forces.” 

Integrated 

•  “...the  major  fault  with  most  of  the  COAs  was  that  we  used  too  many  forces  and  ran  the  risk  of 
leaving  our  ‘main’  tasks  exposed  “ 
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Collaboration  Survey  -  Distributed  Teamwork 

Monitoring  each  other’s  actions  as  a  distributed  team  was  more  difficult 
than  monitoring  each  other’s  actions  in  a  co-located  team. 


Question  3.1 


□  Disagree 
■  Neutral 

□  Agree 
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Question  3.1 

Traditional 

•  “In  the  Traditional  process  a  lot  of  work  goes  on  in  the  background  as  the  lead  nation  develops 
the  COA.  Distributed  team  lead  can  pick  and  choose  the  injects  it  wishes  from  the 
contributing  nations  potentially  avoiding  the  problems.  If  the  Planning  teams  were  physically 
together  this  would  not  occur.” 

•  “SIMPLY  CANT  BEAT  FACE  TO  FACE  FOR  ANY  DECISION  PROCESS.” 

•  “We  seemed  to  know  what  was  going  on” 

•  “GROOVE  functioned  well  in  text  chat.  If  lead  nation  acted  as  a  good  chairman  and  keep 
other  informed  as  to  where  documents  were  located,  planning  procedures,  requirements  and 
objectives  the  MA  and  COA  development  went  well.” 

Integrated 

•  “Although  the  system  allowed  some  monitoring  it  required  others  to  keep  briefing  ‘offline’ 
discussions.” 

•  “Sufficient  discipline  was  evident  during  the  experiment  so  that  there  was  good  situational 
awareness  among  the  (wider)  team.  This  could  easily  be  compromised  if  discipline  to  ensure  all 
were  aware  lapsed.” 

•  “You  cannot  see  what  the  other  person  is  doing  when  he  is  not  talking  or  typing  on  the 
computer” 
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Collaboration  Survey  -  Distributed  Teamwork 

It  was  more  difficult  for  distributed  team  members  to  provide  constructive 
criticism/  feedback  on  each  other's  performance,  in  comparison  to  co-located 
team  members. 


Question  32 


T  ( 15) 
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Question  3.2 

The  spread  of  responses  was  comparable  for  the  two  teams. 

Traditional 

•  “Might  be  the  case  generally,  however,  in  MN  LOE  I,  all  inputs  were  dealt  with  and 
addressed...” 

•  “Location  did  not  hinder  opportunities  to  challenge  inputs,  in  fact  I  was  impressed  with  the 
candor  which  was  used  all  around.” 

•  “I  can  argue  in  chat  just  as  hard  as  in  person” 

Integrated 

•^Comments  were  freely  made,  debated,  resolved  (or  otherwise  agreed  to  be  left  unresolved)” 

•  “Face  to  face  it  is  a  lot  easier  to  enforce  your  view  and  discuss  differences,  if  that  view  is  at 
the  end  of  a  phone  on  the  other  side  of  the  world  it  is  a  lot  easier  to  ignore.” 

•  “As  long  as  the  feedback  was  accepted  then  it  was  used”  “ 
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Collaboration  Survey  -  Distributed  Teamwork 


It  was  more  difficult  for  distributed  team  members  to  identify  when  other 
team  members  needed  assistance,  in  comparison  to  co-located  team 
members. 
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Question  3.3 

To  put  this  response  in  perspective  one  needs  to  consider  ‘down  time’  during  the  LOE  due  to 
network  problems. 

Traditional 

•  “It  takes  longer  to  detect  stress  by  reading  text  than  by  observing  an  adjacent  co-worker” 

•  “All  one  had  to  do  was  ask  for  assistance  and  it  was  provided.” 


Integrated 

•  “Distributed  teams  rely  on  honesty  to  call  for  help” 

•  “We  asked  each  for  help  if  we  need  it” 
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Collaboration  Survey  -  Distributed  Teamwork 


Distributed  team  members  experienced  more  communication  problems  (e.g. 
misunderstandings,  poor  information  passage)  in  comparison  to  a  co-located 
team. 


Question  3.4 


□  Disagree 

□  Neutral 

□  Agree 


%  of  Responses 
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Question  3.4 

For  both  teams,  some  of  the  ‘Agree”  responses  reflected  concerns  about  technical  communications  (i.e. 
network  problems). 

Since  some  of  the  respondents  did  not  provide  insightful  comments,  it  is  difficult  to  distinguish  technical 
vice  personnel  issues.  Regardless  of  the  reason,  a  larger  %  of  ‘Traditional’  respondents  indicated  they 
experienced  more  communications  problems  as  compared  to  the  Integrated  team. 

Traditional 

•  “Despite  the  occasional  misunderstanding,  problems  were  resolved.  “ 

•  “Without  links  such  as  video  conferencing  it  is  much  harder  to  continue  to  reinforce  a  point  of 
disagreement,  in  the  end  it  is  easier  to  allow  the  point  to  go. . .” 

Integrated 

•  “The  problems  were  limited,  as  described  above.  The  free  flow  of  chat  allowed  for  clarification  questions 
and  they  were  used  .” 

•  “LIMITED  DIFFERENCES  ONLY.  COMMS  PROBLEMS  WERE  NOT  RELATED  TO  HUMAN 
BEINGS  BUT  TO  A  SOMETIMES  NONWORKING  WAN  SYS” 
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Collaboration  Survey  -  Distributed  Teamwork 

It  was  more  difficult  for  distributed  team  members  to  develop  and  maintain 
situational  awareness  of  the  bigger  picture,  in  comparison  to  a  co-located 
team. 
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Question  3.5 

For  both  teams  a  significant  %  of  respondents  disagreed  with  this  statement. 

Traditional 

•  “It  appeared  that  most  participants  were  able  to  develop  and  maintain  situational  awareness 
throughout  the  period  of  the  experiment.” 

•  “Within  the  simple  scenarios  exercised  situational  awareness  was  not  an  issue.  The  greater 
complexity  a  scenario  the  greater  the  chance  of  a  loss  of  information.” 

•  “It  will  always  be  harder  when  you  are  not  co-located,  as  different  people  interpret 
information  in  different  ways,  when  co-located  this  may  be  picked  up,  when  distributed  these 
differences  are  often  missed” 

Integrated 

•  “  Situational  awareness  seemed  to  be  about  equal  among  team  members  as  long  as  the 
communications  pipes  were  up  and  running.” 

•  “OF  COURSE!  IF  YOU  CAN'T  HEAR  THE  COMMENTS  OR  DISCUSSIONS  OF  THE 
OTHER  TEAM  MEMBERS  THERE  IS  ALWAYS  A  LACK  OF  INFORMATION.” 
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Collaboration  Survey  -  Distributed  Teamwork 

It  was  more  difficult  for  distributed  team  members  to  anticipate  the  needs  and 
requirements  of  their  fellow  members,  in  comparison  to  members  of  a  co-located 
team. 
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Question  3.6 

This  question  appears  to  be  very  similar  to  Question  3.3  yet  the  responses  for  both  teams  were 
quite  different  from  the  responses  to  3.3.  This  questions  yielded  the  largest  %  of  ‘Neutrals’.  In 
some  cases  the  comments  did  not  agree  with  the  responses  (e.g.,  some  ‘Neutrals’  appeared  to 
actually  agree  with  the  statement.) 

Traditional 

•  “With  the  situational  awareness  that  occurred  most  participants  were  able  to  identify  gaps  in 
requirements  of  other  teams  and  offer  comment  /suggestions.” 

•  “It  will  always  be  harder  when  you  are  not  co-located,  as  different  people  interpret 
information  in  different  ways,  when  co-located  this  may  be  picked  up,  when  distributed  these 
differences  are  often  missed.” 

Integrated  (‘Disagrees’  did  not  provide  comments.) 

•  “This  was  correct  early  on,  but  as  we  became  more  used  to  each  other  and  the  requirements  of 
the  scenario  and  player’s  positions  it  became  relatively  easy  to  anticipate  fellow  members 
needs” 

•  “Not  as  long  as  the  communication  system  was  operational.  No  difficulties,  in  fact  as  the 
LOE  progressed  the  ability  to  anticipate  each  others  needs  improve  as  the  relationships  grew. 
Functionally  we  used  the  same  planners  for  plans,  intel,  and  logistics/force  structure  for  each 
country.  This  was  not  the  design  it  just  evolved  and  proved  to  increase  the  skills  and  efficiency 
of  the  planning  group.” 
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Collaboration  Survey  -  Distributed  Teamwork 


Distributed  team  members  need  to  possess,  or  develop,  different  interpersonal 
skills  to  those  of  a  co-located  team  in  order  for  the  team  to  perform  well. 


Question  3.7 
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■Neutral 

□Agree 
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Question  3.7 

Both  teams  demonstrated  some  ambivalence  on  this  issue. 

Traditional 

•  “Essential  that  personalities  are  not  allowed  to  interfere.  Inter-Team  building  chat  required  at 
times.: 

•  “The  interpersonal  skills  used  are  very  similar,  it  is  just  more  difficult  to  interact  with  anybody 
you  are  not  co-located  with” 

•  “Must  be  polite  and  patient.  Must  remember  that  there  are  differences  in  the  way  we  conduct 
staff  planning.  Must  remember  to  not  use  colloquialisms  and  acronyms  when  working  with  nations 
who  speak  English  as  a  second  language.  “ 

Integrated 

•  “Clearly,  without  the  benefit  of  seeing  body  language  the  power  of  either  the  verbal  or  written 
word  increases.  This  requires  and  increased  awareness/sensitivity  by  all  participants.” 

•  “Distributed  team  members  need  to  possess  the  ability  to  communicate  both  written  and  orally. 
Without  the  face  to  face  interchange,  distributed  members  cannot  rely  on  body  language  to  assist  in 
communicating.  This  did  not  arise  as  an  issue  during  this  LOE.” 

•  “Same  skills  necessary,  just  a  bit  more  difficult  without  the  body  language,  etc.” 
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Collaboration  Survey  -  Distributed  Teamwork 


It  was  more  difficult  to  reach  consensus  over  decisions  in  a  distributed  team. 
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Question  3.8 

The  majority  of  both  teams  indicated  that  it  was  not  more  difficult  to  reach  consensus  in  a 
distributed  environment. 

Traditional  (Most  of  the  “Agrees”  did  not  provide  comments.) 

•  “The  lead  nation  set  the  rules!” 

•  “No  more  difficult  that  a  co-located  team.  Our  team  had  just  as  many  consensus  issues  as  the 
broader  team.” 

•  “FOR  TRADITIONAL  PLANNING  WHEN  DISTRIBUTED  NEED  TO  ESTABLISH  A 
FORMAL  PROCEDURE  FOR  EACH  PLANNING  STEP  TO  REACH  CONSENSUS-  ON 
WHAT  BASIS  ?-  WHO  IS  THE  BOSS  /  LEAD  AND  WHAT  DOES  THAT  MEAN  ?-  WHO 
HAS  MOST  INFLUENCE  ?-  IS  CONSENSUS  REALLY  REQUIRED  ON  ALL  ISSUES  ?- 
WHAT  HAPPENS  IF  NO  CONSENSUS  ?-  ETC” 

Integrated 

•  “Keep  in  mind  we  are  not  trying  to  achieve  consensus. . .” 

•  “Not  difficult  at  all,  easier  in  fact.  It  is  easier  to  concur  than  to  be  present  and  see  the 
disagreement  in  a  face.” 

•  “Not  really.  All  of  us  are  used  to  using  radio  communications  and  email  as  part  of  our 
decision  making  process.  “ 
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General  LOE  Comments 


•  More  tool  training  needed 

•  Vignettes  not  sufficiently  challenging 

•  Inadequate  IPB  information  provided  with  vignettes 
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Majority  of  LOE-related  comments  (i.e.,  tools,  vignettes)  were  provided  by  the  Traditional 
team. 

Come  comments  regarding  the  Integrated  process: 

•  “I  really  doubt  that  real-life  planning  in  an  Integrated  Planning  Team  as  practiced  here  is  a 
way  nations  will  accept  to  do  business.  Either  you  need  high-ranking  representatives  in  the 
team  who  have  the  vote  to  decide  on  behalf  of  their  nation  (questionable  as  far  as,  for  instance, 
force  generation  is  concerned)  or  you  end  up  in  a  kind  of  sequential  planning  again.  “ 

•  Another  comment  offered  by  a  Traditional  participant: 

“In  addition,  the  United  States  has  got  to  fix  the  Operational  Assessment  Process.  The  Allies 
need  to  have  access  to  ONA  or  MN  RDO  will  not  happen.” 

•  “FOR  ME  THE  WORKING/PLANNING  IN  THE  INTERACTIVE  PROCESS  HAD  BEEN 
A  VERY  BIG  AND  POSITIVE  EXPERIENCE.” 

•  “Overall  I  was  surprised  at  how  effective  the  interactive  planning  was.  I  am  not  sure  if  the 
COAs  we  delivered  were  detailed  enough  but  I  certainly  feel  that  they  had  ownership  from  all 
nations  at  the  time  of  delivery.” 
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Summary  Observations 

•  No  strong  trends  were  observed  though  the  teams  did  differ  on 
some  questions. 

•  Traditional  team  noted  more  issues  related  to  doctrine  and 
cultural  differences  than  did  the  Integrated  team 

•  Traditional  team  was  somewhat  less  positive  or  found  it  more 
difficult  to  evaluate  the  effectiveness  of  the  National  and 
Coalition  cells,  as  compared  to  the  Integrated  team. 

•  Integrated  team  found  it  easier  to  assess  the  value  of  their 
contributions  (given  distributed  environment)  than  did  Traditional 
team 

•  Teams  had  common  criticisms  of  the  COAs. 

•  Both  teams  had  significant  majorities  that  felt  that  it  was  not 
more  difficult  to  reach  consensus  in  a  distributed  environment. 
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Analysis  Workshop  Feedback 

This  appendix  presents  the  voting  results  and  comments  collected  during  the  analysis  workshops. 

Voting  Results:  Examination  of  Process  Characteristics  Identified  for  COI  #1 

Scale:  1  (Very  Similar),  2  (Somewhat  Similar),  3  (Different),  4  (Very  Different),  5  Unknown 
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6.  Effectiveness  Of  Inter-Country  Interactions 
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7.  Effectiveness  Of  Intra-Country  Interactions 


8.  Balance  In  Skills/Experience/Background  of  Teams 
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1.  Session  1:  Introduction  and  Overarching  Questions 

Ann  Arnold 
JHU/APL 
240-228-6639 
ann.arnold@jhuapl.edu 
APL  PM  {#9} 

Christine  Salamacha 

JHU/APL 

443  778  4976 

Christine.  Salamacha@JHUAPL.EDU 
Analysis  Coordinator  {#10} 

Grace  DiPietro 
APL 

443-778-8617 
grace.dipietro@jhuapl.edu 
analyst  {#11} 

Kendall  Wheaton 

Canadian  Forces  Experimentation  Centre 
613  991  6151 
wheaton.kr@forces.ca 
Observer  {#12} 

David  Connell  Operations  Research  Analyst 
Canadian  Forces  Experimentation  Centre 
613-991-6141 
connell.d@forces.ca 

Canadian  Observer  Cell  (Groove  evaluation)  {#13} 

Minor  comment  regarding  the  overarching  questions:  Will  the  analysis  try  to  answer  the  question  -  Which 
planning  process  was  more  effective?  or  alternatively  Which  planning  process  is  more  effective  for 
particular  scenarios?  Perhaps  the  graybeard  session  will  address  this.  {#24} 

Marvin  Barnes  ,  Branch  Chief,  S&T  Liaison 
JFCOM  (J9) 

757-836-2834 
barnesm@je.jfcom.mil 
UK  Control  {#14} 

Christian  Jahnke 

Bundeswehr  Operations  Command  (BwOpsCom),  Potsdam 

+49  3327  50  2533 

ChristianlJahnke@bwb.org 

white  cell  coordinator  at  Potsdam  {#15} 
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Keith  Curtis 
USJFCOM  J9/MITRE 
757-836-3724 
curtisk@je.jfcom.mil 
kcurtis@mitre.org 
J9  LOE  Director  {#16} 

Wing  Commander  Davie  Paton. 

SOI  Battlelabs  within  the  UK  Ministry  of  Defence. 

Tel  No  (+44)207  218  9938. 

E-mail  at  djwcbmblabs@btopenworld.com. 

UK  white  cell  and  experiment  sponsor  for  the  UK  {#17} 

Lieutenant  Colonel  Mark  L  Jackson 

United  Kingdom  Liaison  Officer  to  United  States  Joint  Forces  Command 
757  686  7742 

jackson_gbr@jwfc.jfcom.mil 

Observer  in  anticipation  of  participating  in  MN  LOE  2  in  Nov  02.  {#18} 

Chuck  Sinex 
APL 

240-228-5617 
c.sinex@jhuapl.edu  {#19} 

Aus  wants  to  know  did  the  exp.  work?  Were  we  able  to  conduct  distributive  planning?  Session  I 
overarching  Questions  {#20} 

Was  the  experiment  a  success?  Were  we  able  to  conduct  international  distributive  planning  {#22} 

Yes  re  #20.  We  started  with  some  fairly  skeptical  people  and,  at  the  end,  1  found  that  they  were  converted 
to  what  we  were  tiying  to  do.  But  the  proof  will  be  -  did  the  COAs  that  were  produced  meet  the  mail 

{#23} 

Jeffrey  Acosta 

NATO  Concept  Development  and  Experimentation  (CDE)  Cell  (FIN-53) 

Supreme  Allied  Commander  Atlantic  (SACLANT) 

DSN:  488-3687/COMM:  (757)  445-3687 
acosta@saclant.nato.int  {#196} 

2.  Session  2:  Process  Characterization  Data 

The  dynamism  in  the  vignettes  was  something  that  we  sought  to  achieve  by  not  constraining  the  players 
with  either  the  hardware  or  software  that  they  could  use  {#25} 

When  characterizing  the  process,  it  would  be  useful  to  characterize  each  phase  with  the  mode  used  (e.g., 
national  or  all  hands)  for  each  vignette.  This  would  be  a  simple  matrix  with  phases  across  the  top,  vignettes 
listed  down  the  side  with  the  mode  used  in  the  block.  {#26} 

The  cultural  aspects  may  deserve  future  investigation  {#27} 

National  Staff  Training  and  processes  vary;  these  may  be  reflected  more  in  the  Traditional  analysis.  The 
consequence  of  developing  an  integrated  MN  staff  process  would  be  the  need  to  create  a  common  training 
process  across  all  nations.  This  would  have  national  and  inter  service  cultural  implications.  {#28} 

Was  the  move  within  the  traditional  team  to  a  single  workspace  vice  seperate  countries  done  to  alleviate 
groove  problems  or  to  speed  up  the  planning  process.  Note  that  this  also  allowed  them  to  evolve  into  more 
of  an  integrated  planning  process.  {#29} 
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UK  team  struggled  with  trying  to  follow  Lead  Nation  concepts.  {#30} 

Brit  Traditional  Team  developed  MA  template  that  tended  to  be  used  from  that  point  on.  {#31} 

UK  template  for  traditional  side  needs  to  be  explored.  Refinement  of  this  could  be  done.  {#32} 

Traditional  team  learned  from  early  play.  Integrated  team  did  not  evolve  and  try  to  improve  on  process. 

{#33} 

Traditional  team  -  virtual  workspace  by  talking  with  national  cell  partners.  {#34} 

GE  forced  to  use  just  one  virtual  workspace  because  of  network  problems  {#35} 

Traditional  tended  to  put  in  more  refined  products  into  chat.  {#36} 

Did  constraints  of  the  network  restrict  traditional  development  and  cause  more  of  a  traditional  approach 
since  it  was  difficult  to  converse  with  Australia  and  Germany  early  on.  Did  this  form  the  traditional 
approach  at  an  early  stage?  Was  it  more  of  a  "self  reliance"  which  formed  the  traditional  approach?  {#37} 

UK  found  it  easier  to  track  what  was  happening  with  Traditional  team  rather  than  Integrated  team.  {#38} 

Chat  text  -  not  the  same  info  on  the  screen  at  the  same  time.  {#39} 

A  planning  cell  located  in  one  nation  manned  by  people  from  that  nation  all  imbued  with  that  nation’s 
culture  and  training  background  will  instinctively  think  alike  and  act  together.  Would  it  be  useful  to  make  a 
comparison  with  a  genuinely  MN  organization  by  studying  a  planning  cycle  conducted  by  the  ACE  Rapid 
Reaction  Corps  which  is  manned  by  many  different  nations  across  all  planning/staff  cells?  By  force  of 
necessity,  both  in  peacetime  planning  and  on  operations  in  the  Balkans,  this  headquarters  has  integrated 
different  military  cultures  into  a  single  entity.  {#41} 

Synchronization  problem  made  it  difficult  to  track  chat.  {#42} 

For  next  Experiment  synch  all  computer  clocks  (all  nations  to)  Zulu  24  hr  time  {#43} 

RE#26  Cultural  aspects  are  particularly  evident  in  the  national  planning  concepts.  {#44 } 

Integ  used  audio  -  traditional  did  not  {#45} 

UK  Traditional  didn't  use  audio  because  it  slowed  down  the  system-  used  up  too  much  bandwidth  {#46} 

When  system  may  crash  -  Integ  team  better  placed  to  continue  with  the  plan.  If  you  lost  the  lead  team  in 
Traditional  then  it  was  a  long  time  before  the  lead  could  be  picked  up  {#47} 

Traditional  did  not  use  audio  in  UK.  {#49} 

Would  have  handover  as  shift  changes  occur  during  a  real  multi-day  event  {#50} 

Internal  SA  within  team  was  a  specific  area  that  the  UK  Fluman  Factors  people  were  looking  at=  {#5 1 } 

Although  the  LOE  scenarios  were  of  relatively  short  duration,  it  was  noted  that  in  a  'real  world'  scenario  the 
duration  and  execution  of  an  event  would  take  place  over  days/weeks.  The  traditional  workspace  was 
much  easier  to  acquire  scenario  awareness  versus  the  Integrated  team,  this  has  implications  for  the  times 
when  you  need  to  conduct  handovers  and  bring  in  new  personnel,  the  integrated  team's  workspace  needed 
to  virtually  be  read  in  entirety  to  determine  what  discussions  etc.  had  taken  place  and  what  decisions  were 
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invoked.  The  traditional  workspace  effectively  used  the  discussion  tool,  which  was  very  easy  to  read  into 
at  any  point  in  the  scenario.  {#52} 


A  principle  of  Battle  Procedure  is  Concurrent  Activity;  even  if  comms  are  lost  with  the  Lead  Nation  the 
others  should  continue  to  work.  When  comms  are  re-established  then  a  quick  comparison  of  work  should 
get  all  back  on  track.  {#53} 

Groove  spends  a  lot  of  time  with  synchronization.  This  slows  done  the  system.  If  someone  is  off  line  when 
they  come  back  there  seemed  to  be  system  slow  downs  or  interruptions.  {#54 } 

System  down?  -  Which  copes  better?  Depends  on  time  of  failure  and  whom  you  lose.  {#55} 

In  distributed  environment  which  team  is  more  adversely  affected  by  network  problems.  {#56} 

Traditional  again  used  self-  reliance  because  of  issues  of  connectivity.  Any  conclusion  we  reach  with 
regard  to  multinational  flavor  should  be  tempered  because  of  problems  that  traditional  side  faced  in  shared 
approach.  Often  connectivity  caused  the  traditional  to  work  independently  and  then  collaborate  later.  {#57} 

A  critical  phase  of  any  MN  operation  appears  to  have  been  excluded  from  the  time  analysis.  The  time  spent 
by  military  staff  developing  COAs  may  be  very  short  compared  with  the  time  the  Politicians  will  take 
selecting  the  COA  to  be  adopted  and  giving  the  military  permission  to  then  develop  that  COA.  The  same 
politicians  may  well  then  come  back  and  change  it  -  in  effect  completely  changing  the  COA.  It  is  suggested 
that  future  MN  LOEs  should  include  Political  Advisers  as  players.  {#58} 

Need  to  compare  the  time  allotted  to  each  phase  with  collaboration  tool  performance,  vignette  duration,  and 
previous  vignette  topics.  {#59} 

Planning  time  questions:  do  the  individual  countries  (as  lead)  have  any  trends?  Does  the  fact  that  some 
vignettes  mirror  previous  ones  lead  to  changes  in  the  timings?  Planning  tended  to  be  fit  to  the  time 
available  via  the  schedule  and  not  to  create  a  product  under  a  specific  measurement  factor.  {#60} 

Unused  time  is  difficult  to  capture  -  some  time  was  spent  by  contributing  nations  waiting  for  Lead  to  return, 
provide  guidance  {#61} 

Development  strategies-  (chuck  graphs)  unused  time  could  also  be  how  much  time  players  in  a  supporting 
role  are  not  fully  used,  sitting  around  waiting  for  a  task  {#62} 

Time  is  less  important  than  quality  of  product.  {#63} 

In  regard  to  comment  #58  on  the  Political  dimension  of  COA  development,  I  have  assumed  that  this  part  of 
the  process  was  set  aside  for  purposes  of  conducting  the  experiment  in  a  fixed  period.  Was  this  in  fact  the 
case?  {#64} 

Issue  was  that  planning  was  done  at  operational  rather  than  tactical  level  -  hence  extensive  reearch  was  not 
done. ..not  appropriate  for  the  level  of  planning  was  assumed  {#65} 

Need  warfighting  vignettes  for  the  future  {#66} 

Open  source  searching  was  intended  to  be  used  to  support  the  planning  processes  however;  Autonomy  and 
Team  Brain  were  not  invoked  by  the  participants  to  support  this  objective.  {#67} 

Beware  overclassifying  LOEs  or  saying  that  a  database  is  classified  when  in  fact  all  the  data  in  that 
database  is  open  source  to  the  world  either  through  arms  control  treaties  or  CNN!  {#69} 

Team  Brain  and  autonomy  -  How  much  use  was  made  of  them?  {#70} 

Suggest  that  if  you  provide  a  lot  of  information  about  Pacifica  it  is  important  that  the  teams  get  the  chance 
to  use  what  they  have  learned  prior  to  the  experiment.  {#71 } 
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Value  of  Autonomy  and  Team  brain  called  into  question  {#72} 

Believe  traditional  went  to  a  common  space  because  of  connectivity  issues.  {#73} 
rE  #73  -  AGREED  {#74} 

Based  on  observing  the  Integrated  Team  the  first  week,  it  appears  Groove  dropouts  affected  the  Integrated 
planning  process  schedule  since  the  Lead  nation  (1)  often  waited  until  comms  came  back  to  continue  the 
planning  process  and  (2)  could  not  get  results  back  from  their  foreign  counterparts  who  had  been  requested 
to  supply  specific  planning  components.  {#75} 

Future  LOEs  must  make  it  very  clear  at  which  level  each  Vignette  requires  the  staff  cells  to  work:  Grand 
Strategic,  Strategic  or  Operational.  {#76} 

I  observed  the  Integrated  Team  cells  passing  a  fair  amount  of  planning  information  by  private  mail  during 
the  first  week.  {#77} 

Groove  tool  use  by  traditional  -  high  for  threaded  discussion  and  use  of  whiteboard  with  annotations  was 
very  effective.  {#78} 

The  process  on  LOE  1  asked  the  teams  to  provide  only  1  x  COA.  This  unrealistic;  military  planning  teams 
are  required  to  produce  at  least  2,  usually  3  x  COA  for  the  politicians  to  consider.  They  usually  chose  the 
fourth!  {#79} 

RE#76  in  establishing  the  level  of  planning  for  different  vignettes  cultural  difference  in  the  understanding 
as  to  what  level  of  planning  is  supposed  to  be  done  at  what  level  of  command.  In  case  of  the  LOE  1  it  was 
agreed,  I  believe,  that  we  were  operating  at  the  level  of  MNC.  This  would  in  the  GE  tradition  preclude  any 
tactical  planning.  {#80} 

Integrated  team  was  not  effective  at  tiying  innovative  ways  to  either  use  Groove  or  the  tools  therein. 

Were  the  Groove  trainers  as  swept  up  as  they  should  have  been  and  indeed  did  their  lack  of  experience 
impede  the  use  of  Groove  for  this  exercise?  {#81} 

Canada  is  completing  a  report  on  the  Groove  software  for  use  as  a  collaboration  tool  and  will  forward 
soonest.  {#82} 

Appropriate  mapping  must  be  readily  available  to  the  teams  as  soon  as  they  are  given  the  problems  at 
future  LOEs.  {#135} 

The  baseline  of  information  must  be  even  for  all  the  players  in  future  LOEs.  {#136} 

3.  Session  3:  Vignette  Evaluation  Surveys 

Suggest  that  surveys  be  given  to  participants  partway  through  a  vignette  to  get  feedback  during  the 
exercise.  Most  surveys  at  the  end  were  rapidly  filled  out  in  order  to  end  the  days  work  as  opposed  to 
providing  valid  input.  {#83 } 

Re  #83  Disagree.  Too  disruptive  in  mid-vignette.  During  the  vignette  they  should  be  concentrating  on  the 
Vignette.  UK  players  understood  that  filling  in  the  Questionnaires  were  as  much  a  part  of  the  experiment 
as  running  the  vignette  and  they  stayed  until  the  paperwork  was  finished.  {#84} 

Point  taken,  however  some  mid-scenario  questions  can  be  revealing  to  the  process  and  the  tool  use.  {#85} 

Believe  that  this  best  done  by  Human  Factor  analysts  observing  -  rather  than  the  players  having  to  break  off 
from  their  work  {#87} 
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Although  not  part  of  the  vignettes;  problems  with  the  individual  tools  and/or  the  Groove  interface  were 
handled  independently  and  solutions  to  these  were  not  shared  collaboratively.  I  found  different  ways  to  try 
to  resolve  problems  when  encountered  but  had  no  way  of  sharing  this  knowledge  among  player.  Separate 
Groove  space  for  collaborating  and  sharing  lessons  learned  may  have  been  useful.  {#88} 

Concern  that  there  was  no  attempt  to  ensure  parity  of  skill  or  experience  levels  across  the  lnteg  and  Trad 
teams  internationally.  National  white  cells  would  try  to  make  teams  of  equal  ability  but  no  attempt  made  to 
ensure  that  one  particular  job  specialization  was  not  pre-eminent  in  a  particular  international  team  {#89} 

How  swept  up  were  the  Groove  trainers  with  respect  to  handling  problems  and  using  the  various  tools 
available.  {#90} 

only  1  member  of  ea  UK  team  used  Team  Brain  and  Autonmoy  -  danger  of  drawing  too  many  conclusions 
from  such  a  small  number  {#91} 

1  person  (IT  literate)  can  influence  the  whole  thinking  of  a  national  team  {#92} 
relegated  to  status  of  exhibition  events  {#93 } 

Brain  and  Autonomy  were  not  heavily  utilized  by  the  players  for  various  reasons;  some  participants  did  not 
access  to  these  tools  {#94 } 

Because  BrainEKP  and  Autonomy  were  not  included  in  the  formal  training  prior  to  the  LOE,  most  were  not 
familiar  comfortable  using  these  tools.  {#95} 

Participants  like  Groove  but  there  is  no  comparison  with  any  other  product.  Maybe  they  would  say  this 
about  another  product.  {#96} 

The  slide  on  page  32  about  Lack  of  continuous  audio  being  a  detriment  might  look  very  different  if  it  was 
shown  as  Integrated  vs.  Traditional  response  {#97} 

Vignette  Summary  Observations  stressed  the  importance  of  Leadership.  However,  I  am  concerned  by  some 
of  the  more  detailed  comments  about  the  role  of  the  Lead  Nation.  In  MN  Ops  an  essential  aspect  of,  and 
quality  in  leadership  is  to  bring  your  coalition  partners  along  with  you.  This  involves  consultation, 
discussion  and  sensibility  to  other  nations  concerns.  Dogmatic  direction  by  one  nation  is  not  leadership  - 
rather  it  will  lead  to  the  dissolution  of  a  coalition!  Again  we  are  back  to  training  of  individuals  to 
understand  the  nature  of  MN  operations.  {#98} 

RE#98:  Leadership  in  the  case  of  MN  ops  should  be  exercised  by  giving  subtle  and  sensible  directions, 
rather  than  issuing  stringent  directives  in  order  to  be  able  to  keep  MN  players  on  board.  {#99} 

Analysis  of  requirements  should  be  investigated  more  in  upcoming  LOEs  for  collaborative  tools.  {#100} 

It  is  important  that  the  final  report  accurately  state  the  caveats  surrounding  the  findings  since  the  sample 
size  was  small.  {#101} 

UK  is  promoting  more  involvement  of  political  advisors  in  LOE  II  {#102} 

Canada  would  be  happy  to  see  a  political  dimension  added  to  the  next  LOE  as  well.  This  will  probably 
require  longer  scenarios.  {#103} 

There  must  be  political  scenarios  considered  but  a  roadmap  of  how  to  integrate  this  must  be  laid  out  now. 
{#104} 

For  future  LOE  I  suggest  that  we  look  at  political  involvement  (esp  for  ROE  and  the  other  countries 
involved).  We  need  to  be  able  to  experiment  on  warfighting  issues  and,  by  implication,  this  means  that  we 
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have  to  be  able  to  ran  classified  scenarios  -  should  be  OK  if  we  ensure  that  only  the  MIC  nations  take  part. 
Need  to  be  able  to  do  this  to  maintain  the  credibility  of  the  LOE  (and  its  process),  to  demonstrate  that  we 
are  not  afraid  to  address  the  issues  of  interest  and  to  continue  to  progress  on  issues  of  interest  along  the 
roadtoOC-04.  {#105} 

Ref  #105.  Not  only  participation  but  also  Observer  status  should  be  restricted  to  MIC  nations  only.  {#106} 

4.  Session  4:  Collaboration  and  Distributed  Team  Working  Survey 

All  Traditional  teams  used  physical  whiteboards  -  not  the  case  for  the  Integrated  team  {#107} 

It  is  easy  for  a  small  team  to  integrate;  for  larger  teams  this  will  require  the  creation  and  understanding  of  a 
simple  process.  {#108} 

This  area  was  not  addressed  by  the  Canadian  observer  cell  to  any  extent.  {#109} 

Surveys  need  to  be  simple  and  not  different  surveys.  We  will  do  survey  X  on  Monday  and  survey  y  on 
Tuesday.  {#110} 

Training  needs  to  be  uniform.  If  we  do  surveys  lets  do  a  video  or  something  that  is  uniform  that  trains 
them.  {#111} 

All  surveys  must  be  approved  prior  to  LOE  starting.  {#1 12} 

5.  Session  5:  Graybeard  Evaluations  of  the  COAs 

in  an  actual  incident  you  would  have  a  much  larger  team  working  for  several  days  to  produce  a  COA 
{#113} 

RDO  concept  -  because  you  do  integrated  planning,  do  not  need  more  than  1  COA  {#114} 

Second  bullet,  2nd  point  -  will  also  find  more  than  can  be  put  into  the  COA  {#115} 

US  evaluator  applied  HEAT  tool  to  the  COAs  {#116} 

Headquarters  Effectiveness  Analysis  Tool  (HEAT)  {#117} 

Currently  the  accepted  doctrine  of  all  MIC  nations  is  to  present  a  number  of  COAs  (normally  3)  to  the 
senior  officers.  If  the  RDO  concept  of  integrated  planning  will  only  result  in  a  single  COA  is  to  become 
official  Doctrine  for  one  nation,  this  will  represent  a  fundamental  change  in  MN  interoperability  which  will 
cause  considerable  discussions  in  the  other  countries.  We  will  also  have  to  acknowledge  that  politicians  do 
not  liked  to  be  backed  into  a  corner  by  the  military  by  being  given  only  one  option!  {#118} 

If  completeness  was  the  primary  measure  of  effectiveness,  did  the  Integrated  team  receive  more  positive 
votes  due  to  the  COA  development  as  a  team  vs  the  Traditional  approach  which  restricted  input,  or  rather 
filtered  it  through  the  lead  nation.  {#119} 

Perhaps  there  should  be  a  review  of  the  COAs  to  see  if  there  is  in  fact  more  detail  in  the  Int  team  versus  the 
Trad  team's  products.  This  might  explain  the  lopsided  score,  in  the  face  of  comments  that  the  COAs  were 
fairly  equal.  {#120} 

Does  a  particular  way  to  derive  a  COA  ie  Integ  or  Trad  suit  a  particular  problem  {#121} 

Is  cultural  a  deciding  factor  in  choosing  a  planning  method  {#122} 

Are  there  other  MOEs  to  look  at  for  comparing  the  2  processes  besides  the  overall  scores  from  the 
Graybeard's?  {#123} 
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What  factors  in  HEAT  caused  US  evaluator  to  choose  Integrated  every  time?  {#124  J 

The  point  was  raised  that  the  processes  may  be  more  effective  in  specific  operations  -  perhaps  such  as  the 
humanitarian  operations  considered  in  the  LOE  {#125} 

There  may  be  a  circular  process  with  HEAT.  If  the  J9  participants  are  familiar  with  the  factors  in  HEAT, 
they  may  have  developed  their  integrated  plan  to  address  those  factors.  Since  the  US  J9  players  led  the 
Integrated  team  for  the  entire  first  week,  this  could  have  set  a  strong  bias  toward  generating  a  plan  that 
would  score  high  in  HEAT.  {#126} 

The  vignettes  were  reasonably  similar  and  therefore  was  the  integrated  team  better  at  coming  up  with  a 
COA  given  the  restricted  (low  intensity)  operations.  Some  of  the  questions  to  answer  are:  which  process  is 
better  when  planning  timing  is  short?  Which  process  is  better  for  what  type  of  scenario?  What  process  is 
better  suited  for  particular  scenarios  based  upon  the  level  of  intensity?  {#127} 

Should  a  weighted  factor  system  be  used  to  evaluate  the  traditional  and  integrated  planning  process?  We 
still  don't  understand  why  there  was  a  2.5  to  1  ratio  for  integrated.  Anyone  looking  at  this  will  think 
Integrated  is  statistically  better.  The  sample  sizes  are  small  and  two  graybeards  did  not  complete  the 
process.  {#128} 

show  Graybeards  the  summary  and  get  comments  {#129} 

How  much  did  the  Multinationality  of  the  operation  weigh  upon  the  Graybeards'  selection  of  the  preferred 
COA?  {#130} 

The  raw  data  comments  are  not  strongly  in  support  of  the  graphical  results  obtained,  the  subjective  nature 
of  the  Graybeard  decision  has  not  been  captured  to  support  the  results.  The  low  sample  size  and  the 
unknown  variance  further  jeopardize  relying  on  the  graphical  results  solely.  {#131} 

Keith  (j)  offered  to  provide  the  summaries  from  the  graybeards  and  get  their  feedback  {#132} 

What  are  the  characteristics  of  the  COAs  for  vignettes  E  and  H  that  all  the  graybeards  chose  B  as  superior, 
or  at  least  equal.  {#133} 

Another  option  is  to  resubmit  the  package  to  a  separate  group  for  secondary  (confirmatory)  analysis. 

{#134} 

6.  Session  6:  Summary 

Comment  on  the  proportion  of  time  spent  on  MA  vs  COA  between  the  two  teams.  First,  the  total  time  was 
fixed.  Second,  collaboration  is  an  N  squared  problem.  So  here  is  a  hypothesis:  with  a  larger  virtual  team 
doing  MA,  it  would  be  expected  that  the  first  step  will  take  the  larger  team  longer,  and  then  they  would 
work  to  complete  the  second  step  in  the  available  time.  {#137} 

There  was  evolution  in  traditional  process.  What  we  saw  was  a  cherry  picking  process  where  those  not  in 
lead  nation  role  got  to  see  what  worked  and  what  didn't  and  then  try  the  elements  that  worked  when  they 
took  the  lead.  {#138} 

Human  Factor  people  saw  a  lot  of  dead  time  for  a  contributing  (non  lead)  nation.  {#139} 

Would  prefer  the  collaboration  software  support  to  be  characterized  as  network  degradation  and  periodic 
loss  had  a  significant  impact  on  Groove  performance.  {#140} 

US  lead  integrated  4  of  7  times.  Does  this  skew  the  results  since  this  was  not  the  same  ratio  for  the 
traditional?  US  only  lead  one  time  in  traditional  so  if  US  had  the  lead  more  times  in  the  integrated  then  they 
could  have  developed  a  better  organizational  and  decision  making  process.  {#141} 
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Ref  #137.  Agreed.  Additionally,  current  UK  training  stresses  that  the  more  thorough  the  MA  the  easier  will 
COAs  fall  out  from  the  deductions.  {#142} 

It  is  difficult  to  rely  on  time -based  measures  of  effectiveness  due  to  the  problems  that  the  experiment  had 
programmed  timings  and  so  teams  tended  to  create  an  agenda  at  the  outset  of  the  experiment  and  more  or 
less  follow  that  agenda.  {#143} 

Concur  with  comment  #142,  further  to  that  1  think  it  likely  that  a  MN  collaboration  will  develop  a  better 
(more  complete)  MA.  {#145} 

Did  the  Graybeards  consider  the  MA  for  each  COA?  {#146} 

Check  the  impact  of  how  many  times  a  nation  has  the  lead  {#147} 

More  dead  time  in  Trad  -  Integ  players  more  involved  for  longer  in  the  planning  process  {#148} 

Two  possible  explanations  for  why  the  Int.  Team's  COAs  were  judged  more  complete:  (1)  the  US  lead 
most  of  the  time,  so  the  other  countries  learned  the  style  early  on  and  could  then  spend  proportionally  more 
time  working  on  MA/COA  development  (see  #141),  and  (2)  the  Traditional  team  had  more  dead  time 
within  the  experimental  timeline,  so  the  Int.  Team  had  a  higher  level  of  manpower  actively  working  the 
problem  (see  #139).  Could  be  some  a  combination  of  both  of  these,  and  other  factors  was  well.  {#149} 

Millennium  Challenge  00  used  Information  Work  Space  (Iws)  -  bandwidth  was  a  problem  with  this 
exercise  too  (they  had  a  lot  of  problems) 

They  also  stopped  using  voice  there  because  it  sucked  up  bandwidth  {#150} 

Traditional  did  not  use  audio:  gap  in  relay  (synchronization),  typing  forced  you  to  engage  your  brain, 
system  all  ensured  that  everyone  got  their  input  in. 

Traditional  team  vetted  material  before  it  was  sent  off  {#151} 

With  typed  (chat,  discussion)  you  had  a  record  of  what  was  said  previously.  {#152} 

Subtleties  of  language  lost  for  those  that  English  is  a  second  language  {#153} 

Ref  #  150,  One  of  the  recommendations  from  the  players  was  that  they  would  prefer  to  use  PowerPoint 
instated  of  the  White  Board;  PPT  is  band  width  hungry.  {#154} 

The  Int.  Team  often  stayed  off  audio  because  of  the  bandwidth  problem;  using  notepad  as  the  alternative. 
{#155} 

One  factor  that  was  different  was  the  use  of  voice.  Traditional  did  not  use  voice.  Integrated  did  use  voice. 
Coidd  this  have  been  a  factor  that  made  the  graybeards  pick  integrated?  Just  looking  for  differences  here. 
Any  comments?  {#156} 

We  need  to  carefully  document  the  attributes  of  both  voice  and  text  chat  and  how  they  were  used  during  the 
vignettes.  This  aspect  of  the  LOE  may  be  the  most  important.  {#157} 

Referring  to  comment  #154,  would  PPT  be  more  bandwidth  economical  if  we  dropped  elaborate 
backgrounds  and  complex  graphics?  Perhaps  PPT  would  be  an  efficient  tool  to  share  drawings  if  we  are 
only  sharing  the  actual  drawing.  {#158} 

In  the  UK  they  had  enough  human  factors  observers  that  the  same  people  could  observe  the  same  team 
throughout.  It  allowed  them  to  get  a  better  feel  for  what  was  going  on  in  that  team.  {#159} 
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Some  comments  regarding  the  use  of  voice:  when  utilized  it  tended  to  decrease  the  level  of  SA  because 
you  will  miss  the  talk  if  not  in  the  proper  workspace,  you  are  busy  with  something  else  at  the  time,  you 
have  muted  your  speaker  because  you  are  briefing  someone  or  are  on  the  phone.  Lastly,  since  there  is  no 
record  of  the  conversation,  you  are  not  sure  if  everyone  received  the  message.  As  an  outside  observer,  the 
text  chat  allowed  for  backtracking  to  review  prior  interactions  and  then  allowed  you  to  respond.  {#160} 

If  you  really  want  to  understand  how  planning  operates  in  a  distributed  environment  and  how  to  improve  it 
and  what  kinds  of  problems  are  limiting  performance,  SA  of  the  distributed  planning  process  is  a  critical 
factor  that  needs  to  be  considered  in  future  experiments.  It  was  extremely  difficult  in  this  first  LOE  as  an 
observer  at  one  site  to  really  understand  what  was  going  in  virtual  space  between  the  various  country 
participants.  {#161} 

Was  there  more  innovation  coming  out  of  the  traditional  side  than  the  integrated  side.  If  so  do  we  lose  this 
innovation  if  we  go  to  one  approach  for  LOE  2?  {#164} 

Bandwidth  problems  can  be  managed  by  strict  Information  Management  SOPs  {#163} 

(Read  number  162)  We  need  to  separate  Groove  performance  due  to  network  and  hardware  limitations 
from  shortcomings  of  Groove  tools  like  notepad  jumping  when  a  new  entry  was  made.  {#165} 

The  AUS  integrated  team  started  out  with  {#166} 

(Ref  #162  and  164)  What  is  the  real  question  that  needs  to  be  answered?  Which  is  a  better  planning 
process  or  can  we  in  fact  conduct  MN  collaboration?  In  my  view  we  have  demonstrated  that  we  can  do  MN 
collaboration  and  the  use  of  Groove  contributed  to  this.  Therefore,  I  would  support  our  analysis  of  the 
more  innovative  process,  i.e.  Integrated,  for  LOE  2.  The  traditional  side  is  being  undertaken  now  in  real 
world  operations  but  LOE's  are  intended  to  support  Olympic  Challenge,  which  is  an  RDO  for  the  next 
decade.  {#167} 

Different  -  did  not  evolve  into  the  same  process.  However  -  were  somewhat  hampered  by  the  tool.  {#168} 

Different  -  Traditional  took  on  more  work  level  and  incorporated  at  various  levels.  Different  from  the 
strategy  of  the  Integrated  team. 

Same  -  more  alike  rather  than  each  other  rather  than  like  classical  Traditional  process.  {#169} 

Tool  allowed  streamlining  of  process  -  produced  one  good  COA  rather  than  expend  resources  on  several 
COAs.  {#170} 

Is  it  true  that  the  collaboration  infrastructure  permits  a  single  COA?  {#171} 

Re  170  graybeards  wanted  more  than  1  COA  from  each  team.  {#172} 

Carry  LOE  planning  through  to  developing  OPPLAN  {#173} 

On  the  question  "did  the  Trad  and  Int  evolve  into  comparable  ...  or  significantly  different  processes", 
having  observed  the  manual  traditional  process,  these  2  distributed  processes  are  much  different.  They  both 
involved  MN  collaboration  and  being  distributed  were  very  efficient.  I  thought  they  were  therefore  more 
comparable  in  that  context.  {#174} 

RE  173  -  but  are  we  getting  something  bigger  than  a  LOE?  {#175} 

OPORD  LOE  3  {#176} 

A  thought  is  to  use  MN  LOE  2  is  a  "dress  rehearsal"  for  Olympic  Vision  2003  where  we  could  go  through 
the  entire  planning  process.  {#177} 
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Ref  #173.  Recommend  that  LOE  2  should  have  fewer  Vignettes  to  allow  the  Teams  to  go  further  down  the 
Planning  Cycle.  {#178} 


Vignette  development  scenarios  can  be  adapted  to  exercise  the  whole  of  the  planning  process  by  creating  a 
set  of  three,  one  set  focused  on  the  initial  planning  through  to  COA  development,  the  next  piece  can 
exercise  the  intel  -  polmil  and  the  last  focused  on  the  logistics  and  Opplan  formulation.  {#179} 

One  approach  to  MN  LOE2  is  to  ask  what  are  the  outstanding  questions  we  still  have  about  doing  MN 
operations  in  a  distributed  environment,  then  decide  which  of  those  questions  are  best  addressed  in  MN 
LOE  2.  {#180} 

Should  we  have  a  complete  planning  process  for  LOE  2?  Should  we  bring  in  experts  who  work  this 
process?  Should  we  bring  in  polical  members  for  each  team?  How  many  vignettes  would  we  be  able  to  do? 
{#181} 

The  real  challenge  will  be  to  compare  and  contrast  the  two  processes.  I  believe  we  have  to  do  both  to 
present  a  balanced,  objective  view.  {#182} 

Another  alternative  for  LOE  2  could  be  simultaneous  or  overlapping  vignettes,  which  are  more  realistic.  I 
would  also  recommend  that  nations  be  given  different  timelines  to  conduct  their  planning  (ie.  6,  8  or  10 
hour  duration)  in  order  to  evaluate  the  question.  When  time  is  short  which  planning  process  is  more 
effective?  {#183} 

Integrated  -  LN  less  of  a  leader  rather  than  a  manager.  Also  was  the  lead  most  often  for  the  Integrated 
process.  {#184} 

The  LOE  only  examined  MA  and  COA  development  of  the  staff  planning  process.  These  are  the  simple 
parts  of  the  staff  planning  process.  The  rest  of  the  process  leading  to  the  operation  order  with  "Red 
Teaming"  is  where  staff  planning  becomes  difficult.  For  example  there  was  no  logistics  and  strategic 
movement  play.  This  often  results  in  a  COA  being  discarded.  As  a  result  once  a  COA  is  decided  upon, 
you  MAY  see  the  traditional  use  "functional"  planning,  then  revert  to  "traditional"  when  the  staff  prepares 
to  publish,  distribute,  and  execute  the  order.  The  next  two  LOEs  need  to  examine  this  with  a  "traditional" 
and  "functional"  staff  as  was  used  during  the  first  LOE.  {#185} 

Groove  predicting  30%  reduction  in  bandwidth  with  version  2  {#187} 


7.  LOE  2 

Review  Classification  of  the  LOE  {#191} 

MIC  nations  only  as  players  and  Observers  -  no  outsiders.  {#193} 

Make  very  clear  what  level  teams  are  to  work:  Grand  Strategic,  Strategic,  Operational  (will  never  be 
tactical)  {#195} 

Groove  needs  to  have  a  much  better  and  more  efficient  data  capture/storage/retrieval  capability  for  the 
analysts  in  the  future  if  we  really  want  to  understand  how  the  distributed  process  operated. 

Start  with  a  clear  page  of  the  spaces  each  vignette 

Information  Manager  for  the  LOE 

Make  spaces  identifiable  as  new  spaces  {#198} 

Need  a  much  better  data  capture  and  retrieval  capability  in  Groove  for  the  next  LOE  if  we  are  going  to 
really  be  able  to  reconstruct  and  understand  the  distributed  process.  {#199} 
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Dates  of  planning  conference  are  February  13-14  {#200} 

There  is  more  than  mapping  needed.  Need  to  review  RFIs  and  other  request  to  determine  what  scenario 
information  should  be  available.  This  would  cut  down  some  of  the  RFIs.  {#201 } 

JHU/APL  brief  for  planning  conference  required. 

Main  metrics  for  LOE  II  -  situational  awareness. 

Three  options:  (1)  only  open  source  (2)  limited  release  of  information  (3)  significant/extensive  release 

{#202} 

Need  dedicated  observers  in  each  sites  {#203} 

Control  group  will  have  full  awareness,  vignettes  will  be  used,  and  surveys  will  be  used  to  measure 
situational  awareness  {#204 } 

There  were  significant  op  tempo  issues  during  LOE  I  {#205} 

If  Classification  is  put  at  NATO  Secret  this  will  incorporate  5  out  of  the  6  MIC  nations.  There  may  well 
also  be  a  bilateral  between  US  and  AUS,  or  an  ABCA  agreement,  which  would  enable  AUS  to  join  in  at 
NATO  Secret.  {#206} 

Would  like  to  run  LOE  II  on  CFBLNet  {#207} 

During  each  vignette  the  level  of  planning  must  remain  at  a  constant  level.  {#208} 

See  comments  under  session  6  regarding  the  vignette  design  and  options  to  consider.  {#209} 

Need  sufficient  level  of  complexity,  warfighting  so  that  planners  do  not  need  to  go  down  to  tactical  level  to 
be  kept  sufficiently  engaged.  {#210} 

Way  to  make  it  complex-  make  it  more  real-  political  issues  etc.  {#211 } 

Vignettes  should  be  complex,  warfighting,  and  classified.  {#212} 

Need  to  script  the  vignettes,  requires  more  preparation  {#213} 

Nations  need  to  ensure  that  appropriately  trained  officers  are  participating  in  the  LOE  {#214} 

Do  we  need  a  CVAT,  Red  Team  or  OPFOR  to  support  this  LOE?  {#215} 

Need  someone  to  play  to  role  of  Commander  during  the  LOE  -  someone  to  answer  to  -  will  provide 
feedback  {#216} 

could  we  get  flag  rank  officer?  {#217} 

Need  to  articulate  the  staff  skill  set  as  an  outcome  from  the  development  conference.  {#218} 

Flag  ranked  NATO  officers  {#219} 

Skill  or  experience  -  Set  prerequisites  and  all  abide  by  these.  {#220} 

Focus  of  LOE  II  will  be  on  the  development  or  use  of  the  ONA  {#221 } 

BEWARE!  LOE  means  EXPERIMENT  not  Exercise!  As  more  and  more  demands  are  placed  upon  the 
construct  of  LOE  2  it  seems  to  sound  more  and  more  like  an  exercise  planning  conference.  {#222} 
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If  SA  is  the  focus  of  MN  LOE2,  perhaps  a  longer,  multi-day  vignette  which  progressively  builds  the 
elements  of  an  ONA  might  be  a  way  to  get  a  handle  on  how  SA  develops,  where  the  problems  are,  and  the 
impact  of  varying  levels  of  classified  data  that  can  be  released  to  the  participants.  {#223} 

We  had  too  many  resources  in  LOE  1  -  there  was  no  challenge  in  solving  a  problem  with  limited  resources. 
{#224} 

Do  we  continue  to  have  two  concurrent  processes?  {#225} 

Look  at  JWID02  tools  for  ONA  support  during  MN  LOE  2.  {#226} 

Sub  working  groups  for  planning  conf  for  LOE  2  {#227} 

End  of  1st  Planning  Conf  will  have  requirements  for  nat  reps  to  take  back  to  countries  to  fill  {#228} 

We  must  focus  on  what  essentially  it  is  that  we  want  as  our  test  objective.  Is  to  compare  two  planning 
processes,  or  is  it  to  demonstrate  MN  Collaboration  with  emphasis  on  SA,  ONA  and  information  sharing? 
Remember  that  a  traditional/integrated  approach  is  possible  within  the  various  country  planning  cells. 
{#229} 

We  also  need  to  assess  what  each  country  can  contribute  in  terms  of  the  planning  and  execution  of  the  LOE 
2.  The  answer  may  lead  us  to  determine  the  exact  scope  that  we  adopt.  Lastly,  we  need  to  tailor  LOE  and 
Olympic  Challenge  events  towards  the  long-term  goals  of  this  experimentation  effort.  {#230} 

Data  extraction  must  be  built  into  the  timeline  during  the  running  of  the  LOE  {#23 1 } 

Need  to  upfront  coordinate  analysis  activities;  need  Analysis  Working  group  established  {#232} 

Rather  than  two  teams  using  different  planning  processes,  perhaps  one  uses  two  teams  with  different  levels 
of  classified  information  and  compare  the  impact  on  the  ONA  SAs  produced  by  the  two  teams. 
Alternatively,  one  might  use  one  team  that  starts  with  no  classified  data,  then  add  some  classified  data  and 
examine  the  impact  on  the  ONA  SA,  then  add  an  extensive  amount  of  classified  data  and  assess  the  impact 
on  ONA  S A.  {#233} 

Need  the  "traditional"  and  "integrated  teams."  There  is  political  dimension  that  must  be  considered  in  order 
to  convince  multi-national  partners  some  time  down  the  road  that  they  should  spend  the  money,  time  and 
resources  to  adopt  RDO.  In  addition,  the  LOE  must  demonstrate  that  the  integrated  method  is  in  deed  the 
solution.  What  if  a  hybrid  is  a  solution?  What  if  the  traditional  method  can,  with  the  use  of  collaborative 
planning  and  information  sharing  tools,  be  used  to  conduct  RDO?  Finally,  there  may  resistance  to  RDO 
because  it  is  a  U.S.  initiative  which  they  perceive  is  being  forced  on  them.  The  best  way  to  do  this  is  to 
have  the  two  team  provide  a  valid,  empirical  evidence  that  justify  spending  the  time  and  money  to  adapt  the 
technologies,  and  Doctrine,  Tactics,  Techniques,  and  Training  Procedures  required  to  conduct  RDO  on  a 
multinational  level.  {#234} 

Need  integrated  or  joint  analysis  work  g  {#235} 

#235  follow  on  "work  group  that  incorporates  analysis  from  countries."  {#236} 

Need  to  spell  out  the  skills  and  requirements  for  the  observers  who  are  tasked  to  look  at  the  SA  of  the  Nat 
and  Internat  Teams  {#237} 

Suggest  that  we  adopt  an  experimentation  hypothesis  test  versus  a  set  of  test  objectives.  In  this  regard  we 
would  develop  a  hypothesis  (ie.  The  Integrated  Planning  Process  is  more  effective  at  MN  collaboration). 
The  experiment  is  then  designed  to  test  this  hypothesis  through  the  use  of  vignettes  focused  on  CROP  and 
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ONA.  This  approach  may  then  be  used  to  determine  how  we  test  such  as  do  we  need  two  teams  etc..? 
{#238} 


Could  we  get  joint  reservists  for  the  experiment?  JFCOM  has  joint  reserve  unit.  Not  ASDW  guys  (the  1 
weekend  a  month  people).  Easier  to  get,  you  can  train  them  to  support  this  experiment.  Services  pay  for  it 
as  training.  {#239} 

Investigate  the  use  of  JRU  to  support  MN  LOE.  {#240} 

Get  the  schedule  out  early  -  outcome  of  development  conference.  {#24 1 } 
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APPENDIX  F 


MN  LOE  I  Graybeard  Evaluation  Template 


The  questionnaire  consists  of  three  parts:  Part  A,  Part  B  and  Part  C. 

Part  A  asks  you  to  critique  the  two  Courses  of  Action  developed  by  the  planning  teams.  The 
COA  template  provided  to  the  players  is  comprised  of  three  sections:  Section  I:  CONOPS/Tasks, 
Section  II:  Requirements  and  Section  III:  Likelihood  of  Success.  The  COA  template  is  provided 
in  your  package:  MN  LOE  I  COA. doc. 

The  Measures  of  Effectiveness  (MOEs)  to  be  considered  when  evaluating  the  COAs  are 
suitability,  completeness  and  accuracy.  The  critical  questions  to  be  asked  for  the  MOEs  are: 

♦  Suitability:  “Does  the  COA  support  the  Commander’s  intent  and  desired  end-state?  Is  the 
COA  acceptable  to  multinational  partners  and  the  host  country?” 

♦  Completeness:  “Does  the  COA  lack  certain  relevant  and  critical  information?” 

♦  Accuracy:  “Does  the  COA  include  errors  due  to  incorrect  information  or  assumptions?” 

The  following  was  provided  to  the  participants  as  guidance  for  their  review  of  the  COAs. 


MOEs 

</> 

CONOPS/ 

QJ  Tasks 

C 

o 

Q 

E 

o 

o 

^  Requirements 

o 

o 

Likelihood  of 

Success 

Completeness 

Accuracy 

Suitability 

Factors  to  consider  include: 

Omissions  of  critical  actions  required  to 
address  unique  operational  situation, 
critical  data  sources  not  considered, 
specification  of  chronology  of  events, 
etc. 

Factors  to  consider  include: 

Use  of  erroneous  data/assumptions  or 
incorrect  interpretation  of  information  in 
developing  action  plan 

Fa  ctors  to  conside  include: 

Support  of  Commander's  intent, 
support  of  desired  end  state, 
consistency  with  guidance/checklists 
provided  in  Joint  Doctrine  publications, 
etc. 

Factors  to  consider  include: 

Omissions  in  support  requirements, 
critical  data  sources  not  considered 

Factors  to  consider  include: 

Errors  in  determining  impact  of  COA  on 
Pacifica  mission,  Errors  in  assessment 
ofMNF  capabilities/resources,  Errors 
on  assessment  of  host  country 
capabilities/resources 

Factors  to  consider  include: 

Impact  on  ability  to  support  Pacifica 
operation 

Factors  to  consider  include: 

#  of  (known)  risks  not  considered, 
omissions  in  understanding  of  enemy 
vulnerabilities,  omissions  in 
understanding  ofMNF  vulnerabilities 

Fa ctors  to  consider  include : 

Validity  of  assessment  of  host  country 
objectives  and  response 

Factors  to  consider  include: 

Consistency  with  relevant  ROEs  and 
doctrine;  MNF  support  for  the  COA; 
likelihood  of  COA  to  succeed;  Number 
of  constraints  impacting  success  of 
COA;  Flexibility  to  accommodate 
redirection 

Factors  to  Consider  in  Applying  MOEs  to  COA 

Reviewers  are  asked  to  identify  any  and  all  deficiencies,  relative  to  the  specified  MOEs,  noted  in 
each  of  the  three  major  sections  of  the  COA.  The  reviewers  are  also  asked  to  rate  each 
deficiency  using  the  following  scale: 
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♦  1  =  Minor  deficiency;  could  have  some  adverse  impact  on  operational  effectiveness  of  the  COA 

♦  2  =  Moderate  deficiency;  will  likely  have  adverse  impact  on  operational  effectiveness  of  the 
COA 

♦  3  =  Significant  deficiency;  will  definitely  have  adverse  impact  on  operational  effectiveness  of  the 
COA. 


For  each  COA  you  are  provided  three  text  boxes  that  correspond  to  the  three  sections  of  the 
COA.  In  each  text  box  list  the  noted  deficiencies.  Please  label  the  deficiencies  as  a.,  b.,  c.,  etc.. 
For  each  deficiency,  specify  the  applicable  metric(s)  and  assign  a  weight. 

Codes  for  metrics;  C  =  Completeness 

A  =  Accuracy 
S  =  Suitability 

Codes  for  weight;  1,2,3 

Example: 

Section  I:  CONOPS/Tasks 


a.  COA  failed  to  accommodate  for  the  following  activities  related  to  the  evacuation  of 
displaced  persons;.  (C)  (2) 

b. 


Section  II:  Requirements 


a.  COA  did  not  sufficiently  address  requirements  associated  with  in-country  transport  of 
displaced  persons.  (C)  (1) 

b.  COA  has  incorrect  assumptions  regarding  host  country  capabilities  and  resources.  For 
example...  (A)  (2) 
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Section  III:  Likelihood  of  Success 


a.  COA  timeline  too  long  to  satisfy  commander’s  need  in  an  operational  situation  such  as 
this.  (S)  (3) 

b. 


Part  B  asks  you  to  make  a  comparative  assessment  of  the  2  COAs. 

Part  C  provides  space  for  further  comments  you  might  wish  to  make. 

MN  LOE  I  Graybeard  Evaluation:  Identifier  Information 


Completed  by: 

Vignette  ID  (e.g.,  C) 

Thank  you  for  the  taking  the  time  to  complete  this  questionnaire. 
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Part  A:  Course  of  Actions  Evaluations 

TEAM  A: 

Section  I:  CONOPS/Tasks 


Section  II:  Requirements 


Section  III:  Likelihood  of  Success 


F 


TEAM  B: 


Part  B:  Comparative  Assessments 


As  a  commander,  which  COA  would  you  select?  Please  provide  the  rationale  for  your  response. 
If  you  would  not  select  either  COA,  please  indicate  so  and  provide  your  rationale.  If  you 
consider  the  two  COAs  to  be  essentially  comparable,  please  indicate  so  and  provide  your 
rationale. 


PartC:  Other  Comments 


If  there  are  any  other  points  you  would  like  to  raise  about  this  vignette,  either  about  the  planning 
products  or  the  vignette  itself,  please  do  so. 


Once  again  thank  you  for  taking  time  to  complete  this  questionnaire. 
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